FreeFlow now open source

I don’t really have time to maintain it anymore, so I’ve open sourced FreeFlow, my little toolkit and administration tool for Metastorm BPM. Go grab it from CodePlex and make it better!

The problem with flags in Metastorm BPM

Flags are a great way to pass data between processes or to pass data from an external application to Metastorm BPM. There are several ways to raise flags, via the eRaiseFlag executable, using the Raise Flag ActiveX control, via the engine’s XML interface or through the engine’s COM interface. FreeFlow provides a wrapper around the last two approaches. Usage is pretty simple

      Connection conn = new Connection();
      // use to switch between TP and COM
      conn.RaiseFlagBy = RaiseFlagBy.TransactionProtocol;
      conn.RaiseFlag("New Data", new string[] {"some data", "1"});

As an aside, eagle eyed C# coders may be wondering why the last parameter of RaiseFlag doesn’t use the params keyword to simplify usage even further. The problem is there are several overloads of this method (taking user name, password, folder ID etc) so adding params would confuse the compiler since there would be multiple matches for a call to RaiseFlag. One solution would be to give the method a different name but this would make the API less discoverable since different versions of the same method would have different names. Another solution would be to just have one version of the method, with all the required parameters, but that wouldn’t really make life any simpler since the simple usage above would require passing in all parameters. API design isn’t an exact science and sometimes compromises are required.

But back to the main point of this post. Say we are going to create a new folder in Metastorm BPM using the flag data passed to populate the custom variables. Typically this might happen when somebody fills in an online ASP.NET form and we want to kick off some kind of process in Metastorm based on that data. So we’ll have a flagged creation action in the process and add some code to read the data, like so.

%tText:=%Session.FlagData[1]
%iNumber:=%Session.FlagData[2]

Which works fine. OK, say we change the data passed in to

conn.RaiseFlag("New Data", new string[] {"some\tdata", "1"});

Now when we run the code we don’t get a folder created. Instead we get an error in the Designer Log saying “'%inumber' failed while evaluating expression '%iNumber:=data' Error setting value for custom folder field 'inumber'”. This is because flag data passed to the engine is tab-delimited, so if your actual data contains tabs, everything gets screwed up.

We have two problems to solve here. First, how do we handle data with tabs in it, since we probably can’t stop tabs being entered by the user of the ASP.NET form. Secondly, how do we deal with any kind of failure to parse the data passed. This is more of a problem, since currently if we fail to parse the data, we lose it all since the folder never gets created.

So tackling the second problem first, we want to do as little work as possible in the flagged creation action. We will just assign the flag data to a temporary memo variable, since the flag data will not be available outside the flagged action.

%xmData:=%Session.FlagData 

You may find this won’t work for you in earlier versions, at some point only each individual item of flag data could be accessed but it looks to have been fixed in version 7.6. If it doesn’t work, you’ll need to manually combine each piece of flag data.

Next in the Parse conditional action (with no condition), we attempt to assign the flag data to the variables, like so

%tText:=%xmData[1]
%iNumber:=%xmData[2]

This will still fail and will stay at the ‘Got Data’ stage, but at least we haven’t lost the data. The Edit action can then be used to manually fix up the data and get the folder on its way.

So back to the first problem, handling tabs in flag data. Really the only solution to this is to use a different delimiter when raising the flag. None of them are perfect, since potentially any of them could be in the data, but %CHR(160) has worked well for us in the past. Another solution might be to pass your data in some other format such as XML. That will be more complicated but more robust. 

Debugging authentication scripts in Metastorm BPM

I talked previously about how to simplify editing authentication scripts in Metastorm BPM by using the FreeFlow Administrator. What I didn’t talk about was how to debug what is actually going on in your authentication scripts. There isn’t a way to debug these scripts from within Visual Studio, or at least not that I’m aware of, so the only way to debug them is to write some kind of trace statements. This can help clarify which scripts are being executed and what path is being taken through the scripts.

I guess you could just write to a text file but the way I’ve done this in the past is to use the createLogEntry function. This function is provided in most of the authentication scripts supplied by Metastorm and usage of it is as follows.

    var err = new Object; 
    err.eDetectedByMethod = "eLogin()";
    err.eDetails = "logging in via web";
    createLogEntry( err );

This will write an entry to the eLog table. Typically I’ll add something similar to the above code as the first lines in each eLogin method in each authentication script, to see which script is actually being called, then add it where necessary to see why scripts are failing.

Using a different SAP to login to Metastorm BPM via FreeFlow

Something that comes up quite frequently is how to use FreeFlow to login to Metastorm BPM using something other than the default authentication mechanism. This typically happens when SSO has been installed and although it’s possible to use FreeFlow with SSO it can be a hassle to set up, so the user wants to use the standard eUser authentication. This is pretty easy, by using the SAP property, as below. Note, the SAP property is zero based, so if the eUser script is second in your list of authentication scripts, its value needs to be 1.

  class Program
  {
    static void Main(string[] args)
    {
      Connection conn = new Connection();
      conn.HttpServer = "NEWDOOGAL";
      conn.Engine = "NEWDOOGAL";
      conn.ConnectionType = ConnectionType.HTTP;
      conn.SAP = 1;
      conn.LogOn("Doogal", "");
      Console.WriteLine("Session : " + conn.SessionId);
      Console.ReadLine();
    }
  }

There have been some reports of this not working in e-Work version 6. I suspect this is a bug on the Metastorm end, but since version 6 isn’t supported any more, I haven’t investigated too much. I suggest upgrading, it’s not too painful! And if you need help upgrading, you know who to call.

The search for some page rank

I played around with bing today to see if it’s any good. I was pleased to see that a search for Metastorm brought up our forum and the FreeFlow web page on the first page, so clearly bing is a complete success. Or perhaps not, the image search brought up a photo of their former CEO, who hasn’t been there for many years.

But it did make me go off and check the same search against some other search engines (if you try this yourself, you may well get different results, I’d be interested to know if they are wildly different). Yahoo brings up the Process Mapping website on the first page, Cuil has Jerome’s book on the first page and puts the Process Mapping website on the second page and Ask has the forum on the second page (although for some reason they think I’m Dutch).

Finally I checked the other search engine that you may have heard of. OK, it’s the only search engine that matters. And we are way down the search rankings (page 5 for the forums, page 4 for FreeFlow). But the weird thing is if I search on Google’s UK site, FreeFlow and the forums are on the first two pages. That may make some sense for the FreeFlow site since it’s hosted in the UK, but the forums are hosted in Australia. Next I changed my search term to ‘Meta storm’ and was asked ‘Did you mean: Metastorm’, and although all the results were related to Metastorm, both the FreeFlow website and the forums now appeared on the first two pages. We also rank highly on searches for ‘Metastorm development’ and ‘Metastorm consultancy’ so it seems odd that we are so low down on that one search.

So is it a cock-up, conspiracy, some weirdness in the ranking algorithm or have we been marked down for doing something bad? I don’t think we’ve employed any bad practices in our attempts to improve our ranking and if we had, I would imagine that would impact all our rankings, not just on one search, so I think we can eliminate the last option. But I have no idea about the other three.

Editing authentication scripts in Metastorm BPM

  If you’ve ever had problems with SSO in Metastorm BPM, or been required to implement an unusual authentication mechanism, you’ve probably had to go through the jig of removing, editing and re-adding the script through the System Administrator application. Having found this approach less than optimal, I added support for authentication scripts to the FreeFlow Administrator. This works somewhat differently since it allows you to edit scripts directly. This makes the edit, test and debug process much quicker. Edit the script, apply the changes, test the login (the engine picks up the new script without needing a restart) and then debug and start again if necessary. It’s probably not something you’ll use every day but when you do need it, it makes life much easier.

Renaming a user in Metastorm BPM

People change their names for all sorts of reasons, marriage being the most common, but there are plenty of other reasons, such as a change of religious belief or wanting to get rid of an unfortunate surname. Whatever the reason, the Users and Roles utility that ships with Metastorm BPM doesn’t have any support for changing user names. The FreeFlow Administrator can help out here. Just fire it up, change the user’s name and apply the changes. This will change the user name in all relevant Metastorm tables. However it can’t change every reference to that user. For instance if you’ve stored a user name in a custom variable, that won’t get updated, since there isn’t really any way of knowing that the custom variable value refers to a user.

This can also be achieved programmatically. This may be useful if your corporate standards have changed and you need to change every user’s name in the system. Or you may want to prefix the users’ names with the domain name because you’re moving to SSO. Whatever the reason, here’s some sample code to achieve this.

using System;

using FreeFlow.Administration;

namespace ChangeAllUsers
{
  class Program
  {
    static void Main(string[] args)
    {
      Server server = new Server();
      server.Connect("sa", "a", "Metastorm");
      foreach (User user in server.Users)
      {
        Console.WriteLine("Processing " + user.Name);
        user.Name = "domain\\" + user.Name;
        user.ApplyChanges();
      }
    }
  }
}

Documenting Metastorm procedures

You may be aware of the procedure documenter produced by Process Mapping, the company I work for. If not, have a look. It’s a Designer add-in, which means a menu item is added to the Designer’s Tools menu that kicks off the documenter and generates the HTML documentation. I spent a little time today extracting the HTML generation code from the add-in assembly into a separate assembly. This means that the code can be called from anywhere you like, which leads to the possibility of automating your documentation generation. Combine this with FreeFlow and you can automate the documentation of all your published procedures. The following console application demonstrates how this could be achieved.

using System;
using FreeFlow.Administration;
using ProcessMapping.ProcedureDocumentationGenerator;

namespace DocumentProcedures
{
  class Program
  {
    static void Main(string[] args)
    {
      Server server = new Server();
      server.Connect("sa", "a", "Metastorm");
      foreach (Procedure proc in server.Procedures)
      {
        Console.WriteLine("Processing " + proc.Name);

        string filename = "c:\\temp\\" + proc.Name + ".xep";
        proc.Versions.LatestVersion.SaveToFile(filename);

        string htmlFilename = "c:\\temp\\html\\" + proc.Name + ".html";
        DocumentationGenerator generator = new DocumentationGenerator();
        generator.IncludeMapImages = true;
        generator.Generate(filename, htmlFilename);
      }
    }
  }
}

I believe later versions of SQL Server allow the execution of .NET code from within the database, so I would imagine it is possible to add a trigger to the eProcedure table that kicks off this code whenever a new record is added, so documentation will always be up to date.

Of course the procedure documenter isn’t a silver bullet. To generate useful documentation, some work will be required to ensure the notes in you procedures contain useful information.

Debugging server-side scripts in Metastorm BPM

Debugging server-side scripts in the Metastorm BPM can be difficult. One thing that can help with debugging JScript/VBscript scripts is the FreeFlow Administrator. When an error occurs in your script, generally you’ll be given a line number where the error occurred. This may not relate to a line number in the Designer because scripts are merged together when they get published to the database. The FreeFlow Administrator provides a simple way to view your scripts and see the line numbers and hence track down bugs more easily.

Unfortunately in the world of JScript.NET, when an error occurs you won’t get a line number telling you where it happened. If you want to learn more about debugging .NET code in Metastorm BPM, I would recommend Process Mapping’s .NET course (generally presented by myself).

Saving latest versions of procedures from a Metastorm BPM database

One thing that comes up quite often is how to get all the latest procedures from a Metastorm database. The FreeFlow Administrator makes this easy. Simply load it up, connect to the database and use the ‘Save procedures…’ action. Select the folder to save the procedures to and everything will be saved there, included all libraries.

Like everything else in the FreeFlow Administrator, this can all be achieved programmatically, with something like the following.

      Server server = new Server();
      server.Connect("sa", "a", "Metastorm75");
      server.RetrieveLatestProcedures("c:\\temp");

Why would you want to do this? Perhaps you regularly want to get all the latest versions of the procedures in your Metastorm database before you head out of the office and want a script/console app that does this for you.

Finally, you can also get the latest versions of a specific procedure, using something like this

      server.Procedures["Flight"].Versions.LatestVersion.SaveToFile("c:\\temp\\flight.xep");

Saving all folder attachments from a Metastorm database

A question came up on the Metastorm BPM forums about saving all folder attachments from a Metastorm database. Folder data is saved in a particularly strange format so this wouldn’t be very straightforward but with the FreeFlow .NET library things are much simpler. I thought I’d post it here just to show how easy it is. In fact I may start posting some more simple FreeFlow examples here, rather than posting them as downloads on the FreeFlow site, because it’s much easier than fiddling with ASP.NET pages.

It’s a console application that should work with .NET 2 upwards.

using System;

using FreeFlow.Administration;

namespace GetAllAttachments
{
  class Program
  {
    static void Main(string[] args)
    {
      Server server = new Server();
      server.Connect("sa", "a", "Metastorm75");
      foreach (Map map in server.Maps)
      {
        foreach (Folder folder in map.Folders)
        {
          foreach (Attachment attachment in folder.Attachments)
          {
            string fileName = "C:\\temp\\" + folder.FolderId + attachment.FileName;
            Console.WriteLine("Processing " + fileName);
            attachment.SaveToFile(fileName);
          }
        }
      }
    }
  }
}

Revenue models for free software

I have no ideological belief that software should be free, but it certainly seems to be an increasingly prevalent model for software.  Since I don't run my own software company the code I produce doesn't have pay the bills. But although I enjoy writing it, in an ideal world I'd still like to make some kind of income from my work. Probably my biggest piece of code that I've given away has been the FreeFlow library and Administrator, that can be used with Metastorm's BPM product. I've tried out a few different revenue generation techniques with varying degrees of success.

Sell the source - This was the initial model, let anyone download the binaries but charge for the source. It worked reasonably well for the FreeFlow library, which is used by developers to talk to the Metastorm engine. I guess developers are keen to get hold of the source for libraries they are using, to modify for their own needs and reduce the risk if the developer disappears. Unfortunately Metastorm introduced their own .NET library and as time has gone on there's been a lot more interest in the standalone FreeFlow Administrator application. With this, I suspect less people care about the code since if I disappear it won't actually affect their own applications, so sales of the source code have dropped off.

Website advertising - There's been advertising on the FreeFlow website for a long while and I've made a bit of spending money from it, but that's all. One positive of writing software for a BPM product is that most of the ads are for competing BPM products which seem to earn a good amount of money. The downside is that Metastorm is just one of many BPM companies and BPM is still a fairly niche area so I don't get a massive number of visitors to the site.

Donations - This is my latest attempt at earning a bit of cash. The website and the FreeFlow Administrator now have 'Donation' buttons. So far I've not earned anything from this at all. I'm not hugely surprised. If I'm using a free piece of software I very rarely donate any money. If it's shareware, I'm unlikely to pay for the full version even if it has nag screens (WinZip anyone?). The exception to this is SmartFTP, which I paid for because it needed re-installing every few months and it's actually a nice bit of software.

Ads in the application - I haven't tried this yet and I'm not even sure how ads can be added to a desktop application but it seems to be the path FeedDemon is going down. I may wait and see how well it goes for that software and my own donations before going down the same route. One of the nice advantages of not needing to earn any money from my software is I can try out different models for as long as I want before trying out something else.

A buy out - There has been interest from people wishing to buy FreeFlow off me but there have been no concrete offers as yet. I'd definitely consider a serious offer. Even though I have some emotional attachment to the code I've written I'm not dumb enough to refuse a reasonable amount of money if somebody offered it. Unfortunately as this is a niche product, I can't really see there being a huge number of suitors... 

In conclusion, I think if I was starting from scratch I'd probably try writing software with a potentially wider audience. Something like TimeSnapper, which is of use to almost anybody. And I'd probably try a similar pricing model to their software, a basic free version and a more feature-rich professional version. Or an ad-sponsored free version and a paid-for non-ads version. Now I just have to figure out what that software would be...

Oracle on Windows - the most painful application ever

I'd been putting it off for as long as possible but I finally decided it was time for me to get the FreeFlow Administrator supporting Oracle as well SQL Server. Why did I put it off for so long? Because Oracle on Windows is horrible. I've never used it on Unix but I  assume it must be better than the Windows version, or how have Oracle survived so long? 

SQL Server just works, install it and you've got the database, management tools and drivers installed no problem. Try the same with Oracle and the first problem you've got is that the installer is some weird proprietary Java thing that doesn't behave like a normal Windows installer. Then you have to try and install ODBC and OLE DB drivers. My first installation attempt managed to destroy my database, the second time, after many Google searches, actually worked.

Another nice feature of the Oracle drivers for Windows is they seem to provide new versions every week that fix some bugs and introduce some other bugs, which all adds to the excitement. And how about this, an OLE DB driver is a 200MB download, WTF?

To be fair, some things have improved since my last foray into Oracle development. They now have a free version of their database, Oracle XE, which actually uses a proper MSI installer.  They have a new admin tool, SQL Developer, which does make life easier. Of course it's Java based so the UI is dreadful and there doesn't seem to be a way to create a table space which seems like the first thing you'd want to do but it is a step in the right direction.

So what's the point of this post? None really, except I finally got it all working and I'm feeling quite pleased with myself. Perhaps that's why people use Oracle, because they feel like they've actually achieved something after just installing the damn thing.

Tracking down memory leaks in managed code

I've been spoilt in the past when I've had memory leaks in my applications. I've worked at places where AQTime has been readily available. But I've recently noticed what appeared to be a memory leak in the FreeFlow Administrator and given that it's a free application I can't justify spending cash on a memory profiler. I had a search on the internet for a free memory profiler and downloaded a trial version of .NET Memory Profiler from Scitech. This was pretty sweet but when the trial ran out I was again unable to justify spending money on buying it.

I then thought there might an API available to capture the objects allocated by the .NET runtime but I was unable to find one. I'm guessing there has to be one, since otherwise how would memory profilers work? Perhaps the API is unmanaged or isn't documented, either way I was unable to find it. But what I did find was useful none the less. Shipping with the .NET runtime is a DLL called SOS.DLL that can be used from Visual Studio to debug memory leaks. I won't go into configuring Visual Studio to use the DLL since this post covers this in detail (the post talks about Visual Studio 2005 but it works just as well in 2008). Instead I'll cover the steps needed to track down a memory leak.

First up, set a breakpoint in your app. When you hit the breakpoint and are in the debugger, these are some useful commands that can be typed into the Immediate window.

.load sos - this loads the SOS.DLL so you can now start to use the SOS commands

!DumpHeap -stat - this will show a list of objects created, grouped by their type and ordered by the total amount of memory used by the objects. You need to analyse this list and look for anything that looks suspicious. I generally ignore the .NET types and concentrate on my own types, since it's most likely that these are causing the problems. Also, it's pretty hard to figure out how many .NET type instances would be too many, there may seem to be lots of string objects around but how many should there be? I was suspicious about a class called FolderControl, so decided to dig deeper into the details for that class.

!DumpHeap -type <type name> - this shows the details of each instance of the specified type (you don't need to specify the fully qualified type name, just the class name will do). The most important detail listed here is the address, since this will be used in the next command.

!GCRoot <address> - this looks for references to an object, which can help track down why an object isn't being garbage collected and hence causing a memory leak. I found the output somewhat confusing but I guess it may be useful.

!help - this lists all the commands available. There are many more than the ones mentioned above.

!help <command name> - get more information about a specific command.

In my case, my dynamically created control was hooking into the Application.Idle event and never unhooking the event handler. Since the Application object remains active for the lifetime of the application, my control was never garbage collected. In my experience, a lot of memory leaks in WinForms applications are caused by event handler issues like this.

After all that, I feel like a proper hardcore geek. Time to learn some assembler...

Metastorm BPM 7.6 and Windows Workflow part 3 - Using Visual Studio

In the first two parts of this series I've given a brief introduction to the integration of Windows Workflow into Metastorm BPM 7.6. For this final part I'll be discussing what interests me, being able to execute workflows authored in Visual Studio from Metastorm BPM.

The first question was, is it even possible? There is no UI for publishing workflows to the Metastorm database, other than those authored in the Metastorm WF Composer. But looking at the database, it looked like the only tables that needed populating were eMSWorkflow and eMSWorkflowDefinition. After some playing around I came up with the following code, a simple command-line tool. This is by no means bulletproof, it will fail if the workflow has already been published (the next version of the FreeFlow Administrator will let you delete workflows) and will likely fail for a host of other reasons. I was using it purely as a proof of concept.

using System;
using System.Data;
using System.Data.Odbc;
using System.Diagnostics;
using System.IO;
using System.Reflection;
using System.Workflow.Activities;

namespace PublishWorkflow
{
  class Program
  {
    static void Main(string[] args)
    {
      if (args.Length < 4)
      {
        Console.WriteLine("PublishWorkflow usage:");
        Console.WriteLine("PublishWorkflow [ODBC DSN] [ODBC username] [ODBC password] [Workflow DLL] ");
      }
      else
      {
        string fileName = args[3];

        // get the assembly
        Assembly assembly = Assembly.LoadFrom(fileName);

        // get the workflow type
        Type workflowType = null;
        Type[] types = assembly.GetTypes();
        for (int i=0; i<types.Length; i++)
        {
          if (types[i].IsSubclassOf(typeof(SequentialWorkflowActivity)))
          {
            workflowType = types[i];
          }
        }

        if (workflowType == null)
          throw new Exception("Can't find a workflow in the assembly");

        string connectionString =
          string.Format("UID={0};PWD={1};DSN={2}", args[1], args[2], args[0]);
        using (OdbcConnection connection = new OdbcConnection(connectionString))
        {
          connection.Open();
         
          Guid guid = Guid.NewGuid();

          // write to eMSWorkflow
          using (OdbcCommand command = connection.CreateCommand())
          {
            command.CommandText = string.Format(
              "INSERT INTO eMSWorkflow (eWorkflowName, eWorkflowGuid) VALUES " +
              "('{0}', '{1}')",
              workflowType.FullName, guid.ToString());
            command.ExecuteNonQuery();
          }

          // write to eMSWorkflowDefinition
          using (OdbcCommand command = connection.CreateCommand())
          {
            string fileVersion = FileVersionInfo.GetVersionInfo(assembly.Location).FileVersion.ToString();

            byte[] buffer;
            using (FileStream stream = new FileStream(fileName, FileMode.OpenOrCreate, FileAccess.Read))
            {
              buffer = new byte[stream.Length];
              stream.Read(buffer, 0, Convert.ToInt32(stream.Length));
              stream.Close();
            }

            command.CommandText = string.Format(
              "INSERT INTO eMSWorkflowDefinition (eWorkflowGuid, eWorkflowName, eFileVersion, eFullyQualifiedTypeName, " +
              "eFullyQualifiedAssemblyName, eWorkflowDefinitionType, eRulesDefinitionName, eLoadedTime, " +
              "eWorkflowDefinition, eWorkflowProject) " +
              "VALUES ('{0}', '{1}', '{2}', '{3}', '{4}', 'A', '', GETDATE(), ?, NULL)",
              guid.ToString(), workflowType.FullName, fileVersion, workflowType.AssemblyQualifiedName,
              assembly.FullName);
            OdbcParameter parameter = new OdbcParameter();
            parameter.Value = buffer;
            parameter.DbType = DbType.Binary;
            parameter.Direction = ParameterDirection.Input;
            command.Parameters.Add(parameter);
            command.ExecuteNonQuery();
          }
        }
      }
    }
  }
}

So I published my simple Visual Studio authored workflow using this code and was able to execute it. It would appear the process context activity provided as part of the Metastorm activities isn't required, although I assume it will be needed if you need to get folder information into your workflow.

The next step was to see if I could execute a workflow that contained custom activities. Obviously the problem here is where will the engine pick up the required assemblies (assuming the custom activities are in their own assembly)? I tried putting the DLL in the engine directory, dotnetbin directory and even the System32 directory (since this is where dllhost lives) but none of them seemed to work. All I got was the following useful error message

Microsoft Workflow evaluation (Instance ID 963fc00a-f2f9-40a8-a0df-7111b0706cd7)
For more information refer to the workflow tracking and event tables.

In fact the tracking and event tables contained no useful information. This error message seems to be the default error message when something goes wrong with a workflow and I've never found any useful information in the database. Debugging the engine through Visual Studio seems to be the only way of getting hold of the real error.

In the end the only way I found to get the engine to pick up my custom activity assembly was to install it in the GAC. Once I did that the workflow executed correctly.

So yes it is possible to execute a VS authored workflow in Metastorm, although some frigging around is required. However I have one concern. One problem I encountered whilst researching this was that workflows would abort after a certain time. Once again, the error above was shown in the Designer Log. Debugging the engine showed the problem was the workflow was timing out. I found a registry setting that controls this timeout, which by default is set to 60 seconds. But how does this timeout work? Is a thread kept alive waiting for the workflow to terminate? If so, this will be a problem for scalability if you wish to execute long running workflows. And that is what Windows Workflow is all about, running some code, sleeping for a week, executing some more code etc. I don't have the answer to this question yet, but will investigate further.

Part 1 - The basics

Part 2 - The database tables

Part 4 - Using your own activities

Part 5 - Long running workflows

Part 6 - State machines

Increment by increment

Popular music lyrics don't often use the word "increment" so I was pleased to hear British Sea Power use it on their first album. In fact they use quite a few unusual words and phrases. But "increment" has stuck in my head because it's such an important word to me.

Take programming for instance. Joel Spolsky calls development "The Game of Inches", but personally I prefer to think of it as a game of millimetres. It's probably partly due to my metric background but also inches suggests you might spot you've made progress after a day, whereas often I'll only recognise some progress has been made on a development project after weeks or months. Perhaps I'm just a slower programmer...

The fact is you can't make a lot of headway in a day. Which is probably why agile development has become popular. I'm not sure about parts of agile development but the idea of taking baby steps to get to where you want to be seems like a pretty damn good idea and I've done it myself for years.

Big designs have lots of problems. It's impossible to keep the details of a massive project in your head. Then things change part way through, or the design wasn't as well thought out as you initially thought. Then your boss tells you we need to release something next week... So it goes on.

The Random Pub Finder started as a craply designed site with hardly any content. Six years later, after hundreds of tiny steps, I'm now quite proud of it.

The FreeFlow Administrator started as a simple project to solve a particular problem I had. A year or so later and it's a fully fledged application that beats the Metastorm provided tools in pretty much every respect.

The commonality between these two projects is that both started small and have had working functionality from the start and at all the intermediate steps. Perhaps it's agile but I prefer to think of it as incremental. Agile brings to mind scrum meetings and pair programming. Perhaps they help but they aren't the important bit as far as I'm concerned.

Which makes me wonder why you'd want to develop stuff any other way. Most projects I've seen with huge design documents actually fall into an incremental development lifecycle as they progress anyway, so why not start out that way?