Metastorm BPM 7.6 and Windows Workflow part 5 - Long running workflows

OK, I want to go back to basics for this post. What is the point of Windows Workflow? Here's the first paragraph of the preface to "Essential Windows Workflow Foundation"

"Windows Workflow Foundation is a general-purpose programming framework for creating reactive programs that act in response to stimulus from external entities. The basic characteristics of reactive programs is that they pause during their execution, for unknown amounts of time, awaiting input."

I've highlighted what I consider to be the important point here. A WF workflow may lay dormant for minutes, days or weeks. In the normal world of .NET code, implementing this kind of application can be troublesome. For a start, the application may shutdown and restart before the workflow resumes, so the application needs to cope with this possibility. This means the objects must be persisted and restarted with the correct state. A simple solution like making the executing thread sleep for the required amount of time isn't going to work, in terms of robustness or scalability (imagine 1000s of these workflows being executed at the same time).

So WF helps solves this problem. Without this feature, WF is really just a different approach to producing standard .NET code, either through the designer or generating declarative XML documents to explain how the code should execute. It might well be useful in some scenarios but it isn't really a killer reason to use the technology.

So with that in mind I created a very simple workflow to execute from Metastorm BPM, which you can see on the left. All it does is write to a text file, wait for ten minutes, then write to the text file again.

The first thing to ensure is to set the "Microsoft Workflow timeout (secs)" registry setting to something sensible. Unfortunately setting this to zero doesn't mean no timeout, it just uses the default 120 seconds timeout. So this needs setting to something very large. This is a DWORD value with a maximum value of 4294967295, which I calculate is approximately 49710 days, that hopefully is large enough for most workflows.

Executing the workflow worked fine in this scenario. The workflow idled for 10 minutes then resumed and completed.

So the next experiment was to start the workflow, shut down the engine and then restart the engine and see what happened. I waited... and waited... Nothing. I started another workflow just in case the workflow runtime hadn't been kicked off but still nothing. So there's a bit of a problem there, particularly if you want to run long running workflows.

As an aside, if you're running your workflow asynchronously then you won't be able to use most of the activities provided for interaction with Metastorm BPM. This is a shame. In particular the RaiseFlag activity would be exceedingly useful since it could be used to signal to the parent process that the workflow had completed. As it is, you'll need to do this with the ECL activities provided, which are more complicated to configure or write your own code.

Part 1 - The basics

Part 2 - The database tables

Part 3 - Using Visual Studio

Part 4 - Using your own activities

Part 6 - State machines

Autosizing row heights in a WinForms DataGrid

Autosizing column widths in a WinForms DataGrid is pretty easy, if a little hacky. Although it seems that a similar technique could be used to autosize row heights, the private RowAutoResize method doesn't do what I was expecting it to do. I found some useful code here which I have adapted somewhat so that it can handle DataGrids with IList based classes as their data source. It also looks at all cells to work out the required height.

    public void AutosizeRows()
    {
      int numRows = 0;
      if (DataSource is DataTable)
      {
        numRows = ((DataTable)DataSource).Rows.Count;
      }
      else if (DataSource is IList)
      {
        CurrencyManager manager = (CurrencyManager)BindingContext[DataSource];
        numRows = manager.List.Count;
      }

      using (Graphics g = Graphics.FromHwnd(Handle))
      {
        StringFormat sf = new StringFormat(StringFormat.GenericTypographic);

        // Since DataGridRows[] is not exposed directly by the DataGrid  
        // we use reflection to hack internally to it..
        MethodInfo mi = GetType().GetMethod("get_DataGridRows",
          BindingFlags.Instance | BindingFlags.NonPublic);

        Array dgra = (Array)mi.Invoke(this, null);

        // Convert this to an ArrayList, little bit easier to deal with 
        // that way, plus we can strip out the newrow row. 
        List<object> DataGridRows = new List<object>();
        foreach (object dgrr in dgra)
        {
          if (dgrr.ToString().EndsWith("DataGridRelationshipRow"))
            DataGridRows.Add(dgrr);
        }

        int colCount = TableStyles[0].GridColumnStyles.Count;

        // Now loop through all the rows in the grid 
        for (int i = 0; i < numRows; ++i)
        {
          int maxHeight = 0;

          for (int j = 0; j < colCount; j++)
          {
            SizeF size = g.MeasureString(this[i, j].ToString(), Font, 400, sf);
            int h = Convert.ToInt32(size.Height);

            // Little extra cellpadding space 
            h = h + 8;

            maxHeight = Math.Max(h, maxHeight);
          }

          // Now we pick that row out of the DataGridRows[] Array  
          // that we have and set it's Height property to what we 
          // think it should be. 
          PropertyInfo pi = DataGridRows[i].GetType().GetProperty("Height");
          pi.SetValue(DataGridRows[i], maxHeight, null);
        }
      }
    }

Capital boozing - the Random Pub Finder book

It used to be called vanity publishing. An aspiring author, having been rejected by all the standard publishers, would decide the problem wasn't with themselves and their work, but with the publishers who were clearly blind to the author's genius. So the only solution was to go out and get their work published themselves and sell hundreds of thousands of copies that way. Of course what generally happened was that, after paying out thousands to the vanity publisher, the author would be left with 2472 copies of their masterpiece sat in the loft, after giving out the other 28 copies to friends and family.

Things have changed, we can now all be published authors via the wonderful lulu.com. The beauty of this system is that a forest doesn't get destroyed to sate the author's vanity. Books are only produced if somebody is actually interested in them. And because they are published on demand, books can be updated and corrected quickly and easily.

So to take advantage of this system, we've taken the reviews from the Random Pub Finder and put them all together in one convenient book shaped package. I don't know about you but even with all the latest technologies for reading on a screen, I still much prefer a book. No power problems, no waiting to boot up, no random crashes.

Anyway, if you want to buy a copy, hop over to http://www.lulu.com/content/2003276. I'm still waiting for a design for the cover, but other than that it's all done.

Turn it up to 11

I've just noticed the volume on the BBC media player on their website goes up to 11. I assume the developer of this code is an afficionado of Spinal Tap.

I can't say I've noticed the difference in sound levels between this and other web based apps...

Metastorm BPM 7.6 and Windows Workflow part 4 - Using your own activities

My previous posts on Metastorm/WF integration have had a few comments, so I thought I'd follow up on them.

First up, it is possible to reference your own assemblies containing custom activities in the WF Composer, via the Options dialog that can be found on the main menu. For me this means the Composer could potentially be used by non-technical people, although the workflow designer may still be too difficult to hand over to non-techies (this isn't Metastorm's fault BTW, the design surface is all Microsoft's work).

But before I could check out this feature, I had a problem using the Composer. I haven't used it for a few weeks or months and when I tried to do almost anything I got the following error

Could not load file or assembly 'mscorcfg, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. (mscorlib) 

I suspect this is down to me uninstalling Visual Studio 2005 since I installed the WF Composer which I guess uninstalled the .NET 2.0 SDK which is required by the WF Composer. It's a shame the SDK is required, as it's a 354MB download... If it's just for the one assembly then it's flipping ridiculous.

Anyway, after installing the SDK and reinstalling the Metastorm WF bits, things started working again (the re-install may not be necessary but I uninstalled before I realised what my problem was).

OK, so I thought I'd try to use some of my own activities. My first problem was that any assembly I tried to reference produced the following error

No toolbox items found in assembly!

This was easily solved by adding the following attribute to my activity class definition

[ToolboxItem(true)]

My activities now appear in the toolbox but I'm unable to drag them onto the design surface. I guess I'm missing something in my activity class code but I don't know what. They work in Visual Studio and my own designer so I'm not sure why they fail here. Anyway, although the WF Composer is of passing interest, my main interest is using workflows authored in Visual Studio, so I'm not going to look at it too closely.

Part 1 - The basics

Part 2 - The database tables

Part 3 - Using Visual Studio

Part 5 - Long running workflows

Part 6 - State machines

Complete rewrite - two years on

It's almost two years on since I wrote about one of my previous employers starting a complete rewrite of one of the core pieces of the their application suite. So how have things turned out? To tell you the truth I have no idea, nobody from the company will give me any useful information these days. But one thing that I am sure of, the new software hasn't been released yet. So my back of a fag packet estimate of three and half years development time isn't looking quite as daft as it may have done at the time.

Having said that, it is of course possible that the software is done and the company are just waiting for the right time to release it. I remember being blown away when I discovered that Ultimate Play The Game actually held back the release of the ground-breaking Knight Lore game so they could release the less impressive Sabre Wulf without it looking rubbish.

And if you do know anything about the progress of the software of which I'm talking then post a comment, you can do it anonymously...

Noel Gallagher pushed off stage

I have a theory why the North West has produced more than its fair share of great popular music over the past few years (The Beatles, Joy Division, New Order, The Smiths, Happy Mondays, British Sea Power (although Brighton try to claim them as their own), Elbow, The Fall, The Stone Roses to name but a few). It's the weather. Rain, rain and more rain. People are forced to stay in quite a lot and do something. So a lot of them decide to try their hand at music and some of them succeed. But I would never include Oasis on that list. Although their initial material was catchy, it never really reached the emotional heights that great pop music does. And their output afterwards has been absolutely awful. To be fair, "Wonderwall" could have been a beautiful song had it not been for the vocals of Liam.

Anyway, I'd pretty much forgotten about Oasis, but apparently they are still chugging along with their Beatles parodies. And now we have this wonderful video of Noel being pushed off stage by somebody. Although that's slightly humorous, the bit I love is self-styled tough guy Liam. First he tries to get away from the stage invader, then when the guy has been held down by two or three security guys, he suddenly remembers how hard he really is and walks across to "have a go" at him. I think it really shows how much of a show the whole tough guy image is...

How to kill the hits to your website

I went on holiday a few weeks ago. Unbeknownst to me the Random Pub Finder's database died the day I went away. With no web access I only found out on my return. A few days of emails to our ISP followed (yes we should have had a backup but we didn't; fortunately our ISP did have) and finally the site was back online. A few days after that I went off to Switzerland and what do you know, the database died again. A more paranoid person than myself might think somebody knew I was going away and did it deliberately. I'm not quite that paranoid but am obviously paranoid enough to consider it as a possibility. After the first data loss I'd done a backup, but I didn't have access to it whilst I was out of my office. Anyway, upon my return, the site was restored pretty quickly. So in all the Random Pub finder was offline for about three weeks.

Prior to the database crash, we were averaging about 300-400 visitors a day, now we are barely reaching 100. I guess the broken pages are still in the search engines' caches and who knows if and when we'll get our visitors back...

The wonders of e-ticketing

If you go to Rail Europe and purchase a ticket, chances are you won't actually be sent real tickets. Instead you'll be sent an itinerary with some reference numbers. Then when you get to the station, you go to a machine and pick up the real tickets. So I was trying to get to Basel In Switzerland and chose the train since I had too much baggage to go on the plane and also felt like going on the train was the right thing, environmentally, to do.

So I got to St Pancras and started typing in my reference numbers. My Eurostar tickets popped out but I was unable to get the tickets for the rest of the journey. I thought perhaps this was because I needed to pick them up from the station in Paris so didn't worry too much.

I arrived in Paris, dragged my stuff to Gare d'Est and tried to type in my reference numbers again and got nowhere. The touch screen computer helpfully had an English option with a button so small that it was almost impossible to hit the screen in the correct place to get it working. Nice touch, SNCF. So I started queuing up in the ticket office but time was running out, so I headed for the train. After all, I had an itinerary that included my seat number, so it was obvious that it was my seat. Unfortunately, the ticket inspector didn't see it that way. I didn't have a ticket. Yes, the train was fully booked and nobody was demanding to sit in my seat but I didn't have a ticket. With the man next to me translating, I tried to argue my case, but got nowhere and ended up paying for my ticket again. Ho hum. The ticket inspector did assure me I could get my money back if I went to the ticket office when I got off.

But at least I managed to get to Basel. So the next thing to do was get my ticket refund and get my ticket for the return leg of my journey. Unfortunately the ticket office in Basel informed me I was now in the wrong country to pick up my tickets for the return journey and would have to go to France. Fortunately Basel is only eight minutes from the border so this wasn't too time consuming or expensive. When I got to St Louis, the lady in the ticket office was very nice and her English wasn't too good (although clearly much better than my French) so after getting my return ticket I thought better of trying to explain why she needed to give me some money due to her jobs worth co-worker.

So lesson learned, trains may be more environmentally friendly but train companies don't seem to be in any rush to actually compete with airlines. They cost more, take longer and make life difficult for their customers. To solve one of these problems, here's a suggestion for you, how about sending the tickets along with the itinerary and then I might be tempted to use the train again in Europe?

Babysitting laptops

PCs are so fecking demanding. I've got a few laptops at home ready for some training I'll be doing and every time I switch one of them on I'm assaulted by a bunch of popups telling me I need to update this or that, Windows Update, Java, Macafee, some other crapware installed with Dell. After half an hour of updates, I can finally get round to doing whatever I need to do, which I've forgotten about by now of course.

So how can this be solved? Here's some suggestions

Windows Update - OK, this is probably a necessary evil, although I'm dubious about some of the updates pushed my way. Are they all really necessary?

Macafee - I haven't run a scan for two weeks because the laptop hasn't been switched on at all, I'm not likely to have acquired a virus in that time am I?

Sun Java - Do you really have to keep bugging me to install the Google Toolbar? And what the feck has that got do with Java anyway?

Dell - Stop installing all this crap on your PCs. Offer an option of installing a clean OS, even if it costs a little more. The problem is that all that extra crap you install hits performance and half of it is badly written trash so people start to think Windows is crap and go off and buy a Mac, even though it's actually Dell that's the problem. The irony is that the thing that made PCs successful, open hardware meaning anybody can build them and install whatever they feel like, is the thing that may well kill them off, replaced by the closed Mac...

The Coming Collapse of the Middle Class

This video may be about the US but I'm guessing it also applies to the UK. We may think we are richer than previous generations, but in fact we are spending more on housing, healthcare and have more debt, even though we generally have two parents in work. The problem with having two parents in work is we have no slack in the system. If one parent falls ill or loses their job, we pretty quickly run out of money, since the other parent can't go out to earn extra, they are already in the workforce.

Connecting two PCs with a network cable

Wireless networks are great for home networks, unless you need to copy large amounts of data around. OK, 802.11g is probably pretty fast, but I've got 802.11b here and when I tried to copy 4Gb of data around, things got a bit slow. I had no blank DVDs and my USB drive is only 1Gb so I had to figure out another way to get the data from one machine to the other. So I figured I'd network them the old way, with wires. And here, mostly so I remember how to do it next time, is what's required. This may not all be required but I'm a software man, not hardware, and you know how many software engineers it takes to change a light-bulb right? But given that most of the information I've found on t'internet seems to be "use a cross-over cable", it might be helpful to someone.

First, make sure the two computers are part of the same workgroup. Right click on 'My Computer' and view the computer's properties to see what workgroup the PCs are a member of. If they are on a domain, I guess they need to part of the same domain.

Next, the cable needs to be a cross-over cable. I think I, er, borrowed my cross-over cable from a former employer but they are available from all good computer equipment shops.

Next, make sure TCP/IP is configured to use fixed IP addresses. More often than not TCP/IP is set up to obtain an IP address automatically since they'll get assigned by your ISP or router or some little gremlin who sits in a box on the internet somewhere - look I dunnow how it works. Here your TCP/IP connection is just between two computers, so there is no little gremlin to assign IP addresses. So look at the properties of your network connection and from there look at the properties of the TCP/IP stack. I think you can pretty much choose whatever IP address you like, so long as the two PCs have different IP addresses. I chose 100.100.100.101 and 100.100.100.102 and that worked fine.

Finally, disable your wireless connection on one PC at least. This may well make no difference at all, but it seemed like a good thing to do. If your PC can see another computer via two network connections, how does it choose which connection to use when copying files? Again I don't know, so figured disabling the slow wireless connection would be a good idea.

SonicWall VPN client and wireless networks

I've had this problem for a while. I have a couple of computers connected on my home wireless network which generally can see each other to share files. But sometimes I'm unable to connect to the other machine. I'd always put this down to some weirdness going on in my wireless network and dealt with it by copying files onto a USB drive. But I finally figured it out today. It would appear if I have the SonicWall VPN client running on one of the machines this causes the wireless network to go funny. Shutting SonicWall down fixes the issue.  No idea if this is a bug or if it is required behaviour but now I know why it happens I don't really care.

Cuil not so cuil

It's been done to death already but I can't help agreeing with the pundits who say Cuil will not replace Google as the search engine of choice. I did a search for "Doogal Bell" and most of the returned sites were search listings on other sites that linked to my stuff. OK, I could get to my stuff indirectly this way, but it's hardly what I'm after. Doing a search on Google brought up this site, my home page, the Random Pub Finder and my photos on flickr, which is what I'd expect. It did throw up a few useless directory and search listings sites but further down the list. OK, not very scientific, but it's very rare Google gives me unhelpful results so it's still relevant I think.

The Cuil website says "Rather than rely on superficial popularity metrics, Cuil searches for and ranks pages based on their content and relevance". Isn't this the largely discredited method of searching the web that was used by search engines before Google came along? So how's it going to be better now? It was way too easy to game those search engines by just repeating keywords in the text of your page.

Anyway, it'd be nice if somebody did come up with a decent alternative to Google, it's becoming increasingly obvious that they will soon be a monopoly and we all know what monopolies start getting up to...

Radiohead - Ceremony

One of my favourite bands* covering one of my favourite songs, what's not to like?

And if you like it, download it with KeepVid (this is mostly a reminder for myself, I'm sure you already know about this website since you're probably much more web savvy than me)

* If you've read one of my previous posts, you may think I don't like them anymore, but their last album has mostly restored my faith. As has this cover.

Tracking down memory leaks in managed code

I've been spoilt in the past when I've had memory leaks in my applications. I've worked at places where AQTime has been readily available. But I've recently noticed what appeared to be a memory leak in the FreeFlow Administrator and given that it's a free application I can't justify spending cash on a memory profiler. I had a search on the internet for a free memory profiler and downloaded a trial version of .NET Memory Profiler from Scitech. This was pretty sweet but when the trial ran out I was again unable to justify spending money on buying it.

I then thought there might an API available to capture the objects allocated by the .NET runtime but I was unable to find one. I'm guessing there has to be one, since otherwise how would memory profilers work? Perhaps the API is unmanaged or isn't documented, either way I was unable to find it. But what I did find was useful none the less. Shipping with the .NET runtime is a DLL called SOS.DLL that can be used from Visual Studio to debug memory leaks. I won't go into configuring Visual Studio to use the DLL since this post covers this in detail (the post talks about Visual Studio 2005 but it works just as well in 2008). Instead I'll cover the steps needed to track down a memory leak.

First up, set a breakpoint in your app. When you hit the breakpoint and are in the debugger, these are some useful commands that can be typed into the Immediate window.

.load sos - this loads the SOS.DLL so you can now start to use the SOS commands

!DumpHeap -stat - this will show a list of objects created, grouped by their type and ordered by the total amount of memory used by the objects. You need to analyse this list and look for anything that looks suspicious. I generally ignore the .NET types and concentrate on my own types, since it's most likely that these are causing the problems. Also, it's pretty hard to figure out how many .NET type instances would be too many, there may seem to be lots of string objects around but how many should there be? I was suspicious about a class called FolderControl, so decided to dig deeper into the details for that class.

!DumpHeap -type <type name> - this shows the details of each instance of the specified type (you don't need to specify the fully qualified type name, just the class name will do). The most important detail listed here is the address, since this will be used in the next command.

!GCRoot <address> - this looks for references to an object, which can help track down why an object isn't being garbage collected and hence causing a memory leak. I found the output somewhat confusing but I guess it may be useful.

!help - this lists all the commands available. There are many more than the ones mentioned above.

!help <command name> - get more information about a specific command.

In my case, my dynamically created control was hooking into the Application.Idle event and never unhooking the event handler. Since the Application object remains active for the lifetime of the application, my control was never garbage collected. In my experience, a lot of memory leaks in WinForms applications are caused by event handler issues like this.

After all that, I feel like a proper hardcore geek. Time to learn some assembler...

IE7 vs FireFox : The never ending debate

I was reading a forum today when I spotted yet another debate about FireFox and IE7. Much like the Windows vs Mac vs Linux debate, this is getting incredibly tedious. Both browsers (along with Safari, Opera et al) do a perfectly good job of letting people browse the web. Both have their plus points (e.g. FireFox has better plug-ins, IE tends to work with more sites) so just let it go, will you?

Much like the OS debates, the thing that really gets my goat is the smug superiority of FireFox users. Look, using a minority web browser does not make you a hip happening person, it just means you use a different browser than the majority of people. It doesn't make you a better person just because you're not using software written by the "evil empire".

For the record, I use IE and FireFox and I'm not hugely excited by either...

Zoopla - How much is your house currently worth?

Yet another site for the British obsession with house prices. When I first looked, our house was apparently worth £371,039, now it's worth £375,350, who says house prices are falling? (well me for one...). I'm not sure how valuing it to the nearest pound is possible but hey who am I to argue. Perhaps this is the answer for people having trouble selling their homes, put them on the market for a very exact price, thus showing buyers that you're not messing about and know the value of your house precisely. Well, perhaps.

Any talk of the massive inflation in house prices always makes me think of the quote "House prices are a matter of opinion but debt is real". Even though the value of our house has apparently increased way beyond inflation in the 8 years we have been living here, the debt hasn't reduced dramatically. And due to the way mortgages work, we still have to pay the same amount every month. In fact, come November, we'll probably have to pay a whole lot more. So, why is house price inflation good exactly?

How to escape text in SQL statements

Two things got me thinking about escaping text in SQL statements recently. I had previously thought it was a simple topic that everybody already knew about but given that these two things occurred I can only assume that not everybody knows about why escaping text in SQL is important and how to do it. First people keep ending up on the FreeFlow website because one of the methods in the class library is called SqlEscape, so obviously people want to know how to do it. The implementation is very very simple.

    /// <summary>
    /// Escapes a string so that single quotes are replaced by two quotes for use in SQL expressions.
    /// </summary>
    /// <param name="sql">The string to escape.</param>
    /// <returns>The escaped string</returns>
    public static string SqlEscape(string sql)
    {
      if (sql == null)
        return null;
      return sql.Replace("'", "''");
    }

Second, I got hold of some free PHP code off the web that failed to do any escaping of the text in SQL expressions. I foolishly deployed this to a website without checking the code beforehand. Fortunately I managed to spot the error before any dodgy hacker managed to take advantage of it. In the PHP world, escaping can be done using the mysql_real_escape_string function.

So that covers the how but why is this important? The first, probably least important, reason is that it means queries that include ' in text strings will work. I guess most people come across this problem when they try to enter a name like "O'Connor" into an application and it fails.

The more important issue, particularly with public facing websites, is SQL injection where a hacker can manage to run pretty much any query they like against your database.

There are other ways to solve this problem. Stored procedures and parameterised queries will also do the trick.

The joy of WITH (NOLOCK)

For a long while I thought it odd that reading from a SQL Server database caused the rows that were being read to be locked. Thinking about it for a little longer makes it clear that it's a necessary evil. You don't want some other process to update records as you're reading them, since the results you get back may well be inconsistent. But the thing is this often doesn't matter that much, so long as it's near enough correct. I've only recently come across WITH (NOLOCK) and I think it's wonderful. It's a quick and dirty way to boost the performance of your queries since it means the query can execute without acquiring a lock on the rows you're trying to get hold of.

Opponents of using it will say there are more fundamental problems with your database that could be fixed with some better indices, but sometimes we don't have the time to do proper performance analysis, so a quick fix is welcome. Purists will no doubt look down their noses at using this technique since it's just not the right thing to do. But I don't care, it improves performance for very little cost. Saying that, I do wonder how inconsistent the data may be. For instance, could I read a row that has a half-written text field? I haven't seen anything like this yet but I do wonder if it's possible.