A cross browser XML parser

A post from three years ago detailing how to implement selectSingleNode for XML documents in a cross-browser friendly manner is still getting a good number of hits. Which I guess shows that developers still need to manipulate XML in browsers, even with the increasing popularity of JSON. When we started to rewrite our desktop app on the web, JSON seemed like the obvious choice, but we use XPath in a big way and we wanted our web app to be compatible with our desktop app, so we stuck with XML, which meant having to deal with the different ways XML is supported in different browsers. So we now have a reasonably well featured cross browser XML parser, the source of which you’ll find below.

A couple of things to note, this probably works in IE8 and below but I’ve never tested it since our app needs at least IE9. Also, the code is TypeScript rather than Javascript, since TypeScript is slightly less insane…

// stop TypeScript complaining about stuff we don't have definitions for
interface Window {
  DOMParser;
}
declare var XPathResult;

class XmlWrapper {
  private xmlDoc: any;
  constructor(xml: string) {
    try {
      // try Internet Explorer first. Although later versions have DOMParser, they don't implement evaluate
      this.xmlDoc = new ActiveXObject("Microsoft.XMLDOM");
      this.xmlDoc.async = false;
      this.xmlDoc.loadXML(xml);
      this.xmlDoc.setProperty("SelectionLanguage", "XPath");
    } catch (ex) {
      if (window.DOMParser) {
        var parser = new DOMParser();
        this.xmlDoc = parser.parseFromString(xml, "text/xml");
      } else {
        throw new Error("Can't find an XML parser!");
      }
    }
  }

  public selectSingleElement(xmlNode, elementPath: string): Element {
    return <Element>this.selectSingleNode(xmlNode, elementPath);
  }

  public selectSingleNode(xmlNode, elementPath: string): Node {
    if (xmlNode == null) {
      xmlNode = this.xmlDoc;
    }

    if (this.xmlDoc.evaluate) {
      var doc = xmlNode.ownerDocument;
      if (doc == null) {
        doc = xmlNode;
      }
      var nodes = doc.evaluate(elementPath, xmlNode, null, XPathResult.ANY_TYPE, null);
      var results = nodes.iterateNext();
      return results;
    } else {
      return xmlNode.selectSingleNode(elementPath);
    }
  }

  public selectElements(xmlNode, elementPath: string): Element[] {
    return <Element[]>this.selectNodes(xmlNode, elementPath);
  }

  public selectNodes(xmlNode, elementPath: string): Node[] {
    if (xmlNode == null) {
      xmlNode = this.xmlDoc;
    }

    if (this.xmlDoc.evaluate) {
      var doc = xmlNode.ownerDocument;
      if (doc == null) {
        doc = xmlNode;
      }

      var resultsArray = [];
      var results = doc.evaluate(elementPath, xmlNode, null, XPathResult.ANY_TYPE, null);
      var thisElement = results.iterateNext();
      while (thisElement) {
        resultsArray.push(thisElement);
        thisElement = results.iterateNext();
      }

      return resultsArray;
    } else {
      return xmlNode.selectNodes(elementPath);
    }
  }

  get documentElement(): Element {
    return <Element>(this.xmlDoc.documentElement);
  }

  public xml(): string {
    // for IE
    if (this.xmlDoc.documentElement.xml) {
      return this.xmlDoc.documentElement.xml;
    }

    // Chrome, FireFox
    if (this.xmlDoc.documentElement.outerHTML) {
      return this.xmlDoc.documentElement.outerHTML;
    }

    // Safari
    return (new XMLSerializer()).serializeToString(this.xmlDoc.documentElement);
  }

  public static getNodeText(node: Node): string {
    if (node == null) {
      return "";
    }

    var value: string = node.text;
    if (node.textContent) {
      value = node.textContent;
    }

    return value;
  }

  public static setNodeText(node: Node, value: string): void {
    if (node == null) {
      return;
    }

    if (node.text !== undefined) {
      node.text = value;
    }

    if (node.textContent !== undefined) {
      node.textContent = value;
    }
  }
} 

Google Earth Pro is kind of free

I was excited when Google announced Google Earth Pro was going to become free. A lot of geographical data comes in Shape files, but these aren’t usable in any free applications I know of and can’t really be used on the web without converting to something like KML first. But Google Earth Pro can load up Shape files and convert them to KML/KMZ.

Downloading isn’t a problem but getting hold of a key doesn’t seem to work as far as I can tell. Following the link to grab a free key just keeps redirecting to the download page. So I gave up in the end and grabbed an illicit key instead. Not sure of the legality of that, but if it’s meant to be free and I still need a key and Google won’t give me one, what choice did I have? Anyway a search for “google earth pro serial key or number” gave me a link to a site that produced a key I could use.

UK house price data January 2015

I have uploaded the latest Land Registry house price data to my site. Prices continue their gradual ascent.

The BBC reports the blindingly obvious that there is a wide gap in regional house prices. This is clearly visible if you compare my place of birth with my current abode. Blackburn, like many areas outside the South East, has seen pretty much static house prices since 2008. Kingston upon Thames, like most other areas in London, took a bit of a breather in 2008 and then continued on its upward trajectory. I guess the interesting question is whether this is a permanent change or if the differences in regional prices will revert back to their historical average at some point. I can’t pretend to know the answer to that, but I do know prices in London are insane…

UK Postcode data for February 2015

The ONS have released the latest version of their postcode dataset and I have now uploaded it to my website. A few sanity checks suggest it is OK but let me know if you spot anything strange

How to beat a Strava PR on every ride

I think getting to, or maintaining, a healthy weight is fairly straightforward, in principle at least. Match your calorie intake with your calorie burning. Riders on the Tour de France eat 9000 calories a day, but don’t put on any weight for the obvious reason that they burn through all those calories. Eating less has never appealed to me, so the exercise side of the equation is the one I try to work on and it generally works OK for me.

But motivation can be a problem. Cycling is my exercise of choice and a winter of cold, wet and windy weather can rather reduce the will to get out on the road. I’ve found a few things to motivate me in the past, signing up with Endomondo (and trying to beat various personal bests), buying a new bike and training for the Ride London 100 to name a few.

Strava is the latest motivator. Every ride gives the opportunity to beat a PR on one segment or another, so there are many more chances to get a little boost from receiving a medal at the end of a ride. But sometimes, nothing. A ride of an hour may lead to no achievement.

But there is a way to almost guarantee a PR on every ride. First, sign up with veloviewer. This provides even more geeky information about every segment you’ve ridden. Next, filter the segment list for ones you’ve only ridden once or twice. Next, check the weather to find out the prevailing wind. Now find some segments where the wind will be behind you. Then plan your ride to include those segments. And finally, ride the route!

And voila, chances are you will beat one of your PRs on that ride. And even better, you’ll probably have ridden some new segments during your ride, which will now be in your list of potential PR segments.

I’ve been using this technique for the last few months and I’ve still got 140 segments within 5 miles of my house that I’ve ridden 2 or less times. Of course I’ll get bored of this at some point, but maybe I’ve already found my next motivation tool (aside from Ride London 2015), increasing my Eddington number…

UK house price data for December 2014

I’ve uploaded the latest Land Registry house price data, for December 2014, to my website. Prices seem to be gradually drifting upwards, as do the number of sales.

House price data for November 2014

I’ve uploaded the Land Registry sales data for November 2014 to my site.

Grading bike ride climbs

I’ve updated the route elevation page on doogal.co.uk so you can now grade climbs. It uses the same rating as Strava, which is based on the UCI climb categorization. Hopefully it will be useful if you can’t find a matching segment on Strava.

Non-geographic postcodes added to UK postcode list

I received a complaint that some postcodes were missing from my UK postcode list that were in the original ONS data. I was a little confused by this, since the import is pretty much automated. But looking at my import code I realised I was ignoring postcodes with no location information. I think initially I’d assumed these were postcodes with duff data so ignored them, but for completeness I’ve now added them to the list, adding a few thousand more postcodes. Since they have no associated location data, there isn’t much interesting information available for each of these postcodes but they may be useful for something… An example.

UK house sales October 2014

I’ve just finished uploading the latest Land Registry UK house sales data to my website. I then realised my annual percentage change calculation was wrong and have fixed that.

UK Postcode data November 2014

I’ve uploaded the latest UK postcode data to my site. There seem to be a few changes to the data (some codes for wards and districts have changed and LSOA data is now available for Scotland). I think I’ve found all the anomalies but let me know if you spot anything odd

More options when calculating route elevations

I’ve added a couple of options to my page for calculating route elevations. You can now select options to avoid highways and toll roads so it should be even more useful!

Goodbye Endomondo, hello Strava and VeloViewer

Back in the day I used to go for bike rides just for the fun of it. Then I discovered Endomondo and was able to analyse every part of a ride. And being a geek, that motivated me to ride more, try to ride faster and try to ride further. I could try to beat my best time for 10 miles or distance covered in an hour. But the number of awards was kind of limited, so my motivation started to flag. I got some more enthusiasm by training for RideLondon 100. But once I’d completed that (or to be more accurate, the 86 miles that I was allowed to ride), I was looking for something else to keep me moving, especially now the days are getting shorter and the weather is deteriorating.
I’d heard of Strava before but I’d discounted it because there was no app for my Windows Phone (what’s new right?). But then I realised Endomondo has the ability to export workouts which can then be imported into Strava. So I gave it a try. And I was immediately hooked. Rather than compare activity over a specific distance/time, I can compare efforts over the same bit of road, a segment. So for any ride I might pass through 20 or 30 segments and I generally beat a personal best on at least one of them. And that helps keep me going.
And for the ultimate geek experience, there’s VeloViewer. This runs on top of Strava and gives me an even more in depth view of my performance on segments. It tells me I have cycled on 1472 segments and I have a score of 86.4 (I have no idea if this is good or bad but it’s headed in an upwards direction which seems like a good thing)
VeloViewer was free but is now asking for £9.99 a year, with a pretty limited free version. This seems like a reasonable amount to pay for something so compelling. Strava also has a premium service but I haven’t seen anything in there that I need. My guess is that Strava will probably start to require premium membership to use their API fully which means any VeloViewer users will need to pay for their Strava account to use VeloViewer. I’d have thought that would be a no-brainer for Strava, since all VeloViewer users are the kind of people who are prepared to pay for a service and are therefore likely to cough up for Strava as well.
Anyway, that should keep me going during the long winter months. Maybe my next motivator will be my own little app running on top of Strava…

House price data now with annual change

The Land Registry sales data on my site now includes the annual price change. I thought this would smooth out the seasonal variations but the annual change is still quite volatile. Even so, it may be of interest.

UK house sales September 2014

The latest house price data from the Land Registry is available on my website and prices continue to defy gravity…

Unable to get property 'setState' of undefined or null reference in ckeditor.js

I’ve been working on the latest and greatest Business Optix product which dynamically creates lots and lots of HTML controls, some of which are CKEditor controls. Whilst testing the app and quickly navigating around it, I started getting an error “Unable to get property 'setState' of undefined or null reference” coming from deep in the bowels of CKEditor. Google astonishingly turned up nothing, but I remembered I’ve had trouble in the past with CKEditor when I didn’t explicitly destroy the editors. The fix was simple, like this

for (name in CKEDITOR.instances) {
  CKEDITOR.instances[name].destroy();
}

Property sales data for August 2014

I’ve uploaded the latest Land Registry data to my website. People comparing the data on my site to news stories like this may be wondering why my data shows a new all time average price, whereas the BBC say prices are yet to reach their previous peak of November 2007. I believe this is due to the Land Registry seasonally adjusting their figures whereas the average values on my site uses a geometric mean of the raw data.

UK property sales data for July 2014

The latest Land Registry data for property sales is now uploaded to my site. Not much can be read into a single month’s figures, but the average sale price has reached an all time high, which I guess pleases a government seeking re-election soon and probably dismays first time buyers.

UK Postcodes August 2014

I have uploaded the ONS postcode dataset for August 2014 to doogal.co.uk. Hopefully everything is in order, let me know if you spot any issues.

Land Registry June 2014 data

I’ve uploaded the Land Registry property sales data for June 2014 to doogal.co.uk. Enjoy!

Setting HttpResponse.StatusDescription silently failing

We recently received a complaint from one of our customers. We provide some fairly simple reporting functionality, that allows more technical users to write their own SQL queries. The customer was building a report and when there was a problem with his SQL, sometimes he’d get a detailed error message, but other times he’d get nothing.

When we try to execute a query and an exception is thrown we catch the exception and write the error message to HttpResponse.StatusDescription so it can be displayed in the browser. This can fail if the message is longer than 512 characters long, but this is fairly obvious since setting the StatusDescription property will throw an exception. That clearly wasn’t the problem here.

I finally managed to reproduce the problem but was still confused about what was causing the issue. Then I spotted the difference between the working case and the non-working case. The non-working case contained new line characters. Thinking about it, this was fairly obviously going to be a problem. The HTTP response would not be valid, since the status description header would be split over two lines. But it would certainly be preferable if .NET threw an exception in this case, rather than just silently failing to set the status description.

So our code now looks something like this

        context.Response.StatusCode = 500;
        context.Response.TrySkipIisCustomErrors = true;
        string description = ex.Message.Replace('\n', ' ').Replace('\r', ' ');
        if (description.Length > 512)
          description = description.Substring(0, 512);

        context.Response.StatusDescription = description;
        context.Response.ContentType = "text/plain";
        context.Response.Write("An error occurred - " + ex.Message);

Land Registry May 2014 data

I’ve uploaded the Land Registry data for May 2014 to doogal.co.uk. For all the talk of a house price bubble, the Land Registry data (arguably the most accurate of all the house price indices) doesn’t seem to show much movement at all over the last few months. Of course it’s a different story in London. There was a story about the housing insanity in Hackney a few months ago, and looking at the sales in East London does show prices shooting up over the last year.

Land Registry house price data for April 2014

The Land Registry house price data for April 2014 is now up on my site

ONS postcode data for May 2014 uploaded to doogal.co.uk

I’ve just uploaded the latest postcode data from the ONS to my site. There are over 2.5 million postcodes in there, alive and dead. My data checks suggest everything is in order, but let me know if you find a problem.

Downloading Javascript generated data

I have a number of web pages that generate some data in text areas using Javascript. The only way users could download this data was to copy and paste the contents of these text areas, but I wanted to add a download button to simplify the process. The problem is that this simply isn’t possible in Javascript. The only client-side solutions I’ve seen either require Flash or are not supported in all browsers.

So I came up with a slightly hacky and inefficient solution. The basic idea is to post the data to the server and get the server to return it to the client as a download. The HTML looks like this

      <form action="download.php" method="post">
        <div>
          <input type="hidden" name="fileName" value="locations.csv" />
          <input type="submit" value="Download" />
        </div>
        <textarea id="geocodedPostcodes" style="width:100%;" rows="20" name="data"></textarea>
      </form>

All that is needed is a hidden field that tells the server-side script what the download file name should be and a text area with a name of “data”.

The server-side script is pretty simple, it looks like this

<?php
  header('Content-Disposition: attachment; filename="' . $_POST["fileName"] . '"');

  // add data
  print($_POST["data"]);
?>

All it does is get the requested file name and echo back the data.

It’s seems a bit crazy (and a waste of bandwidth) that this seems to be the only way to achieve a seemingly simple task, but that looks to be the case. I’d be happy to be proved wrong.

Help me go on a bike ride

Last year I saw the various Ride London rides on the telly and rolling through Kingston and fancied doing it myself. Riding round London and Surrey on traffic-free roads is very appealing, compared to the usual stop-start, take your life in your hands experience of cycling round these parts. So the first chance I had, I applied in the ballot for the Ride London-Surrey ballot. And in January I heard I’d missed out on getting a place.

There was one more option. Sign up with a charity and raise some money and get a guaranteed place. So I decided to try and help out Cancer Research. Why Cancer Research? Primarily because cancer affects so many people at all stages of life but also, on a personal level, one of my partner’s best friends lost her life to cancer a couple of years ago, before she reached the age of 40.

I’ve set up a page for donations, added a link from my website and been amazed by the number of people who don’t know me who’ve already donated. If you’ve found this blog or my website useful, or are just feeling generous, then please consider donating some money. I will certainly appreciate it, as will Cancer Research.

Land Registry March 2014 data uploaded

I’ve uploaded the Land Registry house price data for March 2014 to my website. Now that probably all the sales data for 2013 has come in, it’s plain to see sales volumes were up in 2013 and prices continue to drift upwards

The perils of micro-optimisations

A debate has been raging on my website over the use of StringBuilder.AppendFormat in my exception logger code. OK, raging is something of an exaggeration, there have been two comments in two years. But the point made by two people is that rather than

error.AppendLine("Application: " + Application.ProductName);

I should be using

error.AppendFormat("Application: {0}\n", Application.ProductName);

Since this means I wouldn’t be using string concatenation, which is considered bad for performance reasons. My main reason for not doing anything about this is because I’m lazy, but also because the whole point of this code is that it only runs when an exception is thrown, which hopefully is a pretty rare event, so performance is not a major concern.

But then I wondered what the difference in performance is between these two approaches? So I wrote a little test application that looks like this.

    static void Main(string[] args)
    {
      for (int j = 0; j < 10; j++)
      {
        // try using AppendLine
        Console.WriteLine("AppendLine");
        StringBuilder error = new StringBuilder();
        Stopwatch sw = new Stopwatch();
        sw.Start();
        for (int i = 0; i < 1000000; i++)
        {
          error.AppendLine("Application: " + Application.ProductName);
        }
        sw.Stop();
        Console.WriteLine(sw.ElapsedMilliseconds);

        // try using AppendFormat
        Console.WriteLine("AppendFormat");
        error.Clear();

        sw.Restart();
        for (int i = 0; i < 1000000; i++)
        {
          error.AppendFormat("Application: {0}\n", Application.ProductName);
        }
        sw.Stop();
        Console.WriteLine(sw.ElapsedMilliseconds);
      }

      Console.ReadKey();
    }

The results from this app in milliseconds are as follows (reformatted for clarity)

AppendLine	307	315	321	372	394	370	289	298	300	296
AppendFormat	366	360	362	471	353	359	354	365	365	350

So which is quicker? Well it looks like AppendLine might be marginally quicker. But, much more importantly, who the feck cares? We are repeating each operation 1 million times and the time to execute is still less than half a second. Maybe you can pick holes in my test application, but again I would ask who the feck cares? Either approach is really fast.

And this is the main problem with trying to optimise this kind of stuff. We can spend huge amounts of time figuring out if one approach is quicker than another, but a lot of the time is doesn’t matter. Either the code runs quick enough using any sensible approach, or it’s hit so infrequently that even a really poor implementation will work.

Of course we should consider performance whilst writing code, but we should only use particular approaches when we know they are going to produce more performant code. A good example is the StringBuilder class. We can be pretty sure this is going to be better than using string concatenation, otherwise it wouldn’t exist in the first place. That said, if you’re concatenating two strings I really wouldn’t worry about it.

But the key to writing efficient code is to understand what is slow on a computer. Network operations are slow. Disk access is slow. Because of that, anything that requires large amounts of memory (meaning virtual memory i.e. disk access) is slow. Twiddling bits in memory is quick. Fast code is achieved by avoiding the slow stuff and not worrying about the quick stuff.

And once you’ve written your code and found it doesn’t run ask quick as you’d hoped, don’t jump in and replace calls to AppendLine with calls to AppendFormat, profile your application! Every time I profile an application, I’m always amazed at the causes of the performance bottleneck, it’s rarely where I thought it would be.

If you don’t have a profiler, use poor man’s profiling. There are also free profilers available, I quite liked the Eqatec Profiler which seems to be available from various download sites, although it’s no longer available from Eqatec. But whatever you do, don’t get into Cargo Cult Programming

Land Registry data for Feb 2014

I have uploaded the Land Registry house price data to doogal.co.uk.

There’s been a lot of talk in the press recently about there being a two speed housing market, London and the rest of the UK. You can see this illustrated fairly clearly if you first look at house prices in Blackburn then compare them with prices in West London.

Prices in Blackburn can be broken into three distinct periods. Prior to 2003, they were gently rising, probably in line with wage increases. Then in 2003 things went ballistic (I’m not sure of the trigger for that, although I’d guess it was easier access to mortgages). 5 years later in 2008, things ground to a halt, sales fell of a cliff and prices have been flat-lining ever since.

But look at West London and the only similarity is that sales volumes dropped off rapidly in 2008, but you’d never know that there was a financial crisis at all. Houses have been a one way bet for nearly 20 years. You’ve got to wonder how long it can go on.

Bluffers Guide to responsive design

A while back I spent a bit of time making doogal.co.uk more mobile friendly. I’d put it off for a long time primarily because I thought it would be really hard. But actually it turned out to be not too tricky. So here is my not so comprehensive guide to making your site mobile friendly with responsive design

The first thing to do is decide at what screen size your design will change from the normal design to the mobile design. My decision was that tablets should see the standard design but mobile phones should see the mobile design. So the media query I use for all my mobile CSS is

@media handheld, only screen and (max-width: 600px), only screen and (max-device-width: 600px) 

Better menus - My standard menus were tricky to use on a small screen, but fortunately they were built using styled unordered lists i.e. <ul></ul>, so I was able to use the Mobilemenu JavaScript library. This converts the list into a dropdown <select> which is much more useable on mobile devices.

Hide stuff – display:none is your friend. Most websites have bits on the page that may be useful but aren’t entirely essential. On a big screen we can get away with that, but on mobile devices it’s necessary to concentrate on the essential information. So hide anything that isn’t needed. I have several tables with many columns, but each row links through to more information, so I hid several of the columns since the tables didn’t render very well. The best approach to this appears to be CSS like the following

.postcodeTable  td + td { display: none;} 

No more table layouts – We’ve been told for years not to use tables for layout of our websites. But I’ve certainly been guilty of it, since it’s always easier to build a multi-column layout using a table. But now is when the chickens come home to roost. You may want two columns on the desktop, but on a mobile device, you’ll probably want to have the two columns stacked on top of each other, so those tables will need to be converted to div’s which float on the desktop and don’t on mobile devices.

Server side control – The solutions so far have all been client-side. I think this is generally the easiest way to deal with the issue, but there is certainly a good argument for saying content shouldn’t be getting pushed to the client if the client will never actually display it. If you’re using PHP on the back-end, Mobile Detect Library can be used to tailor your HTML before it leaves the server. One place where you may need to do this is adverts. Google’s T&Cs say you can’t hide adverts, so using display:none for them is probably a bad idea.

The beginning of the end of Metastorm BPM

It looks like development of Metastorm BPM has, if not stopped completely, at least slowed down. So I thought I’d write something about my thoughts about what was a big part of my professional career. If you want the full history, this isn’t it, have a look at Jerome’s book.

For me, it all started sometime in 1997. I was writing software for a firm called Bacon and Woodrow that sold actuarial software. It wasn’t really my kind of thing, but it was my first proper job, something to add to the CV. I got a call from a former colleague, Richard Kluczynski, who’d gone off to write his own software, then got a gig with a software house called Sysgenics. Before the days of mobile phones (or at least before I had one), I remember having to wander the streets of Epsom to find a phone box to call him back to discuss properly, away from the office. It sounded interesting but it wasn’t the right time for me as we were trying to get the first Windows version of our software out the door.

A few weeks or months later I got a call from a recruiter asking if I was looking for a new job and telling me about a company called Sysgenics. We were still trying to get our Windows version out the door, but I guess getting called about the same company twice peeked my interest. I remember looking at their website and getting pretty excited about the screenshots of some kind of graphical tool for building workflows called e-work. Before I knew it, I was in Wimbledon for an interview with Steve Brown and Jerome Pearce. And an hour later I was in the pub. This was obviously a great place to work!

And it was. We had no customers but some VC money to keep us ticking over. We were using the latest technologies (Delphi and MS Access!). Having no customers meant we could build stuff and break stuff without having to worry too much about upsetting people using the software, so we were always making a lot of progress.

Before long we were bought out by Metastorm. Looking back, that was actually a bit weird. Initially they seemed to be some massive software company but looking closer, the one product they sold, InForms, was clearly coming towards the end of its life, since it was tied into the dying Novell Groupware. But they had what we needed, money, and I guess we had what they needed, some modern software to sell.

The years flew by, six in fact. By that point we had quite a few customers, the little startup was a proper software house. There was structure and rules, forms to fill in, basically not really my scene anymore. So I flew the nest to work for a financial software house in central London. But a couple of years after that, Jerome asked me to join his little band of Metastorm consultants. So I built a shed and got to work building stuff on top of Metastorm e-work. Metastorm e-work became Metastorm BPM, we carried on calling it e-work… Metastorm started rewriting it from scratch in .NET, releasing it as version 9 (missing out version 8, some kind of off by one error I think).

I’d probably still be working in my shed for Jerome had the financial crisis not hit, caused major stress to our main client who then couldn’t pay us. So I went to Croydon, regretted it almost immediately and started working for myself. Back to the shed…

Then I started doing some work for Steve’s new company, Business Optix. That eventually became a full time job and is where I am now. Meanwhile Metastorm got bought out by OpenText. Given the price they paid, you’d think anyone who’d taken up their share options would have done well out of the deal, but you’d be wrong. Somebody must have made a nice chunk of money out of it, but it wasn’t the people who’d originally developed the software (this isn’t bitterness on my part, I never exercised the option on my shares).

But not content with one BPM tool, OpenText also bought Cordys and Global 360. I guess the writing was on the wall for two of those products at that point, why would a company want three BPM tools? Anyway, it looks like Metastorm BPM is one of the victims. You have to wonder why OpenText bought them in the first place, presumably not for the customer base, already fed up with having to rewrite their processes for version 9, now fuming that they need to rewrite again in some other system.

Land Registry sales data uploaded to doogal.co.uk

I’ve uploaded the latest Land Registry sales data, covering sales for January 2014, to my website. The good or bad news, depending on your point of view, is that prices continue to creep up. 

ONS postcode data for February 2014 available

I’ve just uploaded the latest postcode data from the ONS to my website. My checks suggests everything is OK with the data but let me know if you spot anything awry

Land Registry December 2013 data on doogal.co.uk

I’ve imported the latest Land Registry to doogal.co.uk. As ever, little can be inferred from a single month of data but draw your own conclusions. And let me know if you’d like to see this data in some other way.

doogal.co.uk is more mobile friendly

I’ve been trying to ignore mobile devices for a long time. Although doogal.co.uk has always been available on mobile phones, so long as you were happy to squint and zoom in and out a lot, it’s never been particularly useable. I’d kind of hoped that as more people started browsing the web from their phones, the phones would get bigger screens and I’d get away without doing anything. Whilst some of the phones have got bigger, it seems a lot of people don’t want to talk into a phone the size of a paperback book. Understandable really, they don’t want to look like dicks.

On top of that, Google keeps telling me my site isn’t mobile friendly and it’s got to a point where a significant percentage of my visitors are using small devices to access the site, so I figured it was time to bite the bullet and fix it. So now if you visit from a smart phone, chances are you’ll see the more mobile friendly version of the site. In fact if you squidge your desktop browser to a small enough size, you’ll also see the mobile layout. Of course, just like the regular desktop site, it’s butt ugly but there is a reason why web design doesn’t appear on my CV!

But if you spot anything broken, please let me know in the comments or email.

Improving the performance of MySql bulk inserts

When I was trying to improve the performance of my old server, I came across this page. Unfortunately it didn’t help with the old server and just made me realise I needed to upgrade my server hardware.

But once I got my new hardware, I wanted some tweaks to improve the performance of importing postcode and property sales data. Both were taking days to complete. One suggestion from that page was to use SET autocommit=0;. I was a little sceptical, but was willing to try anything to speed up the import.

So all I did was add the statement on the start of every insert and add a counter to my code. After 1000 inserts, I committed the changes, using the following

              if (count % 1000 == 0)
              {
                command.CommandText = "COMMIT";
                command.ExecuteNonQuery();
              }

And this made a huge difference to my imports. Whereas I was importing for days before, now the time taken was down to hours. Maybe I could improve it further by committing the changes even less frequently, but I’m pretty happy with the current situation.

So in conclusion, if you need your data imports to run quicker, start using SET autocommit=0; now!

November 2013 Land Registry data

I’ve uploaded the latest Land Registry property sales data to my website, covering November 2013. As always, not a lot can be inferred from a single month’s data, but drilling down to individual postcode areas shows fairly different stories for prices and sales over the long term.

Google’s weird support for KML

KML is a standard XML format used to describe geographic data. It’s been owned by Google ever since they bought the company who originally developed Google Earth. So you’d think they’d have great support for KML in their products, right? I thought so as well until the other day when I was asked about a problem with some of the KML from my site. So here’s a summary of how different Google products cope with KML.

Google Earth – As you’d expect, Google Earth has full support for KML

Google Maps Javascript API – Using the KmlLayer class, it’s possible to load up KML files from anywhere on the web. There are some limitations on file sizes and not all of the features of the KML file format are supported, but for simple points and polygons it works perfectly well.

Of course, not everyone is a programmer, so using the Javascript API is not really an option for a lot of people. Unfortunately this is where things get messier…

Custom Maps (Maps Engine) – If you try to create a custom map in the new Google Maps, importing from KML is not even an option (the user interface says ‘Google Engine Lite’, so maybe if you pay some money to Google you’ll be able to import KML), which means you have to switch back to…

Custom Maps (Classic) – This claims to support importing KML. My experiments suggests it copes with KML files containing points (up to a fairly small maximum size), but I couldn’t get it working with files containing polygons.

Fusion Tables – Hidden in a corner of the web, is Google Fusion Tables. Basically it’s a spreadsheet with maps and has always seemed like a cool piece of technology but it hasn’t been pushed by Google much. But it seems Fusion’s support for KML is much better than either of the Custom Maps options. Points and polygons worked for me.

What I find weird is that all the web based technologies above are produced by a single company and all do pretty much the same thing, take a KML file and display it on a map, and yet they all clearly use completely different code. Maybe it’s time for some consolidation there?

Bing Maps – If none of the Google options work for you, maybe Bing Maps will do the trick. Points and polygons seem to work, with a couple of restrictions. There’s a fairly low limit on the maximum file size and the Import button only seems available in Internet Explorer.

No doubt there are other options, so let me know if you find any better alternatives.

Land Registry sales data back to 1995 and a new server

After much fighting with my old server trying to import the 18 million property sales from 1995 to today, I decided it was time to upgrade the server. I’ve now got 4 times as much memory, twice as many CPU cores and I’ve successfully imported all the Land Registry data. I’ve flicked the switch to point doogal.co.uk to the new server and so far everything is looking good. Let me know if you see any problems or have any performance problems. Also let me know if you’d like the property sales data sliced and diced in any other ways.

Handler "PHP55_via_FastCGI" has a bad module "FastCgiModule" in its module list

Trying to get a PHP app working on Windows Server 2012, I got the following error message

“HTTP Error 500.21 - Internal Server Error

Handler "PHP55_via_FastCGI" has a bad module "FastCgiModule" in its module list”

I’d installed PHP via the IIS Web Installer but had failed to do one more thing, enable CGI in ‘Add Roles and Features’ in Server Manager, under Web server (IIS)/Web Server/Application Development. After enabling that, everything sprang to life

The Land Registry destroyed my database

I started to import all the Land Registry data back to 1995 into doogal.co.uk, which was taking a looooong time. So I tried a little optimisation which involved clearing out some of the data I had already imported. Unfortunately that corrupted my database, which meant I had to drop my property sales table and start again from scratch.

So the property sales data on the site has less information than it did, but hopefully it should all be back online in the next few days. 

Kingston Council optimise their revenue capturing techniques

Not content with having one of the highest council taxes in the UK…*

Local authority	2013-14 Average Band D (excluding Parish Precepts)
Gateshead	1,443.20
Rutland UA	1,430.51
Hartlepool UA	1,418.70
Walsall	1,410.26
Nottingham UA	1,404.42
Stockport	1,397.05
Oldham	1,392.95
Kingston-upon-Thames	1,379.65

Not content with charging us £80 a year to park on a street somewhere near our house…

Kingston Council have made some more changes. Now not only can we not park on our own street most of the day, we can’t load there either. Practically I’m not sure how this is meant to work. How are we ever meant to get something delivered? How are older residents meant to get picked up from their house? But not happy with this change, the council have also introduced a parking enforcement smart car that roams the streets trying to catch any parking offenders. This was a big money spinner for Sutton council and I guess it will be the same for Kingston. Last Saturday I was caught unloading my car by this roaming enforcer. Unlike traffic wardens, it’s impossible to have any kind of rational discussion with them since they drive off as soon as they’ve got their photographic evidence.

So what to do? Well Kingston Council maybe behaving like the Goldman Sachs vampire squid, but fortunately there are local elections coming up soon. Frankly anyone who stands in that election who will stand up for residents’ interests will get my vote. Which pretty much excludes any of the three main parties…

And maybe another option is just to not play the game anymore. Get rid of the car, try to reduce my income to a level where I no longer have to pay the full council tax. Or just move somewhere with a more sensible level of council taxation.

Update – I appealed against the ticket since there were no markings on the pavement to show loading and unloading was restricted, and my appeal was accepted. I’m happier, but still not satisfied with the council’s tactics.

*Just look at the original document I pulled this data from to see the dramatic difference in council tax levels, can anyone explain these vast differences?     

Land Registry Oct 2013 data uploaded to doogal.co.uk

I’ve uploaded the latest Land Registry data to my website covering sales in October 2013.

In more interesting news, with a few days to spare the Land Registry have met their pledge to upload all sales data from 1995 to today. That will take a little longer to upload…

November 2013 ONS postcode data uploaded to doogal.co.uk

I’ve uploaded the latest ONS postcode data to my website. My data integrity check suggest it’s all fine but the size of the CSV download has shrunk a bit. I’m not entirely sure why that is, so let me know if you spot any anomalies.

Joining the dots

Today's BBC business pages contain these two headlines
Black Friday flurry for UK retailers
UK household debt hits record high
Black Friday seems to be another pointless tradition we've just picked up from the US, to give us all another chance to buy more shit we don't need and, it would appear, we can't even afford. I will not be participating.

Making cycling safer in London

Cycling is in the news due to several deaths in London. As someone who has cycled quite a lot in and around London (although very little in Central London) I have a few suggestions.

1. Don’t panic. Yes, cyclists have died and that is a tragedy for the families involved. But the number of deaths has remained fairly constant, even with an increasing number of cyclists on the road (see this document for details). And more pedestrians die than cyclists (where is the uproar about that?)

2. Get involved. Cycling may be no more unsafe than in the past but we still compare badly to other countries, even if our own government ministers try to bend this obvious truth (of course Norman Baker is an utter berk, just read his poorly researched conspiracy theory about the death of David Kelly). And pressure from the public may make a difference. Space for Cycling are targeting the upcoming local elections, which seems like a good idea.

Another obvious target for pressure is Boris Johnson. Boris seems to want to be a friend to motorists and to cyclists, which is a noble plan but is doomed to failure. Cars and bikes do not mix, so anything that is going to improve life for cyclists will almost certainly make life worse for cars. So Boris, which side are you on?

3. Help yourself. Here are some things I do when I’m riding my bike

• Find a route that you are comfortable with. One of the weird things about the cycling super highways is they encourage cyclists on to major arterial roads. Generally I try to avoid main roads as much as possible, since they are a pretty unpleasant experience. If you’re commuting to the same location day after day and you don’t feel safe on the route you’re using, then find an alternative. This is is where a smart phone comes in really useful. You can nip down some unfamiliar roads to see if the route is better and if you get lost, you can find your way using the phone’s maps. If the route is better, your phone should be recording where you went.
• Wear a helmet. I’m still surprised when I see a cyclist without a helmet, or even more bizarrely a child with a helmet sharing a bike with a parent without a helmet. It might not help if you get dragged under a lorry, but it will certainly help for less serious accidents.
• Don’t jump red lights. I have to admit I don’t always stick to this, but there are many junctions where you would have to be insane to ignore the lights.
• Assume every driver is an idiot. Most drivers are sensible but there are still plenty who will do daft things, like turning at a junction without indicating, or pulling out right in front of you.
• Don’t undertake lorries. Undertaking in general is not a great idea, but undertaking lorries is an even worse idea since they likely won’t be able to see you at all.

Getting the icon for the user’s default browser in C#

In my last post I showed how to get hold of the associated icon for a file. Now, say you want to get hold of the icon for the user’s default browser, you may think you can just grab the icon for HTML files and that’ll do the job. Whilst that will work for some users, it won’t work for all of them. The application used to open HTML files does not need to be the same application that is used to open websites. For example changing your default browser to Chrome won’t change the default application for HTML files to Chrome. So here’s an extension to that previous code to get the icon for the default browser. It won’t work for XP since the way default applications are handled has changed but it should work with all later OSes.

    private static string GetDefaultBrowserPath()
    {
      const string userChoice = @"Software\Microsoft\Windows\Shell\Associations\UrlAssociations\http\UserChoice";
      using (RegistryKey userChoiceKey = Registry.CurrentUser.OpenSubKey(userChoice))
      {
        if (userChoiceKey != null)
        {
          object progIdValue = userChoiceKey.GetValue("Progid");
          if (progIdValue != null)
          {
            string progId = progIdValue.ToString();
            const string exeSuffix = ".exe";
            string progIdPath = progId + @"\shell\open\command";
            using (RegistryKey pathKey = Registry.ClassesRoot.OpenSubKey(progIdPath))
            {
              if (pathKey != null)
              {
                string path = pathKey.GetValue(null).ToString().ToLower().Replace("\"", "");
                if (!path.EndsWith(exeSuffix))
                {
                  path = path.Substring(0, path.LastIndexOf(exeSuffix, StringComparison.Ordinal) + exeSuffix.Length);
                }
                return path;
              }
            }
          }
        }
      }

      return null;
    }

    public static Icon GetDefaultBrowserLargeIcon()
    {
      string browserPath = GetDefaultBrowserPath();

      if (!string.IsNullOrEmpty(browserPath))
        return GetLargeIcon(browserPath);

      // last chance (probably XP), just grab the icon for HTML files
      return GetLargeIcon("test.html");
    }

    public static Icon GetDefaultBrowserSmallIcon()
    {
      string browserPath = GetDefaultBrowserPath();

      if (!string.IsNullOrEmpty(browserPath))
        return GetSmallIcon(browserPath);

      // last chance (probably XP), just grab the icon for HTML files
      return GetSmallIcon("test.html");
    }

Getting the associated icon for a file in C#

Sometimes it’s useful to display an icon for a file in an application and probably the best icon to display is whatever the operating system uses. The Windows API provides the SHGetFileInfo function for this purpose, so here’s a little wrapper around it. All the methods take a file name parameter, but the file doesn’t need to actually exist.

using System;
using System.Collections.Generic;
using System.Drawing;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;

namespace TestIcons
{
  public static class ImageUtilities
  {
    [StructLayout(LayoutKind.Sequential)]
    struct SHFILEINFO
    {
      public IntPtr hIcon;
      public IntPtr iIcon;
      public uint dwAttributes;
      [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
      public string szDisplayName;
      [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 80)]
      public string szTypeName;
    };

    static class Win32
    {
      internal const uint SHGFI_ICON = 0x100;
      internal const uint SHGFI_LARGEICON = 0x0; // 'Large icon
      internal const uint SHGFI_SMALLICON = 0x1; // 'Small icon
      internal const uint SHGFI_USEFILEATTRIBUTES = 0x10;
      internal const uint SHGFI_LINKOVERLAY = 0x8000;

      [DllImport("shell32.dll")]
      public static extern IntPtr SHGetFileInfo(string pszPath, uint dwFileAttributes, ref SHFILEINFO psfi, uint cbSizeFileInfo, uint uFlags);

      [DllImport("User32.dll")]
      public static extern int DestroyIcon(IntPtr hIcon);
    }

    public static Icon GetSmallIcon(string fileName)
    {
      return GetIcon(fileName, Win32.SHGFI_SMALLICON);
    }

    public static Icon GetSmallOverlayIcon(string fileName)
    {
      return GetIcon(fileName, Win32.SHGFI_SMALLICON | Win32.SHGFI_LINKOVERLAY);
    }

    public static Icon GetLargeIcon(string fileName)
    {
      return GetIcon(fileName, Win32.SHGFI_LARGEICON);
    }

    private static Icon GetIcon(string fileName, uint flags)
    {
      SHFILEINFO shinfo = new SHFILEINFO();
      IntPtr hImgSmall = Win32.SHGetFileInfo(fileName, 0, ref shinfo, (uint)Marshal.SizeOf(shinfo),
        Win32.SHGFI_ICON | Win32.SHGFI_USEFILEATTRIBUTES | flags);

      if (hImgSmall == IntPtr.Zero)
        return null;

      Icon icon = (Icon)Icon.FromHandle(shinfo.hIcon).Clone();
      Win32.DestroyIcon(shinfo.hIcon);
      return icon;
    }
  }
}

None of our politicians have a clue

So we have a Labour party who are talking about a cost of living crisis. Good on them. Until you hear their main policy, which seems to be fixing the cost of energy. I’m pretty sure the talk of the lights going out are completely alarmist, but I doubt trying to fix prices is going to work, the fact is fossil fuel costs go up and down and no politician can stop that happening.

But Ed is missing the point a little anyway. For most people, energy is a big cost but it’s generally outweighed by something else, housing. We hear a lot about energy price rises, but not so much about rises in rent. And if house prices go up, that’s a good thing apparently? As a home owner, I don’t get it. If I want to move up the ladder, higher prices mean I need a bigger mortgage to move anywhere. First time buyers presumably don’t want higher prices. As a parent, I don’t want housing to be completely unaffordable for my child when she gets to an age when she wants her own place. Yes, there are people who gain from house price rises, but I’m pretty sure they are outnumbered by people who would actually gain from falling prices.

But enough of the Labour party, what have the Conservatives got to offer? Oh yes, Help To Buy, the scheme that seems most people are a little concerned about. Apparently it may cause a new housing bubble. What the ….? We haven’t escaped the last bubble yet. We are still number one in the housing bubble stakes. What George seems to have missed is that the problem with our housing market is not a demand problem, there’s plenty of demand. The problem is not enough houses are being built and the ones that are being built cost too much!

So what’s the solution? First, the British need to get over their obsession with house prices and realise houses are a place to live, not pay for their retirement. Second, we need to radically change our planning laws, so we can build a lot more houses. Third, the government needs to start building some council houses.

The sad thing is none of this looks likely to happen. No political party is going to say “We will build houses in your back yard and let other people do the same and the price of your property will then fall, but in the long run it will be good for the country”

Our dog really hates cats

Eyes and nose removed, most internal organs gone. Fortunately it wasn’t a real cat, just a toy.

Postcode populations

doogal.co.uk has some exciting new postcode data (depending on your definition of exciting…). I’m in the process* of importing population data for each active postcode in England and Wales. This includes the estimated population and number of households and comes from the ONS website based on the Census of 2011, which I’ve just realised only covers England and Wales. Scottish data is a separate download and will be my next task. Northern Ireland doesn’t seem to provide population data per postcode.

The data appears on the information page for each postcode and will become part of the CSV downloads once the import is complete. I could aggregate this data up to postcode district and sector level, so let me know if that would be useful to you.

* A database of 2.5 million postcodes can take some time to update!

More UK parliamentary constituency postcode data on doogal.co.uk

I’ve added a UK parliamentary constituency page to doogal.co.uk for England, Wales and Scotland (Northern Ireland continues to not provide its data under an open license). You can view maps and grab KML and CSV downloads of the postcodes for each constituency.

If you have requests for any different slicing and dicing of this data, let me know.

National Park postcodes

Postcode data can be diced and sliced in lots of different ways. So I’ve sliced the UK postcodes so you can grab data for each National Park. I’ve no idea if this will be useful to anyone, but sometimes the weirdest things turn out to be hugely popular, so here it is

http://www.doogal.co.uk/NationalParks.php

Data is available in CSV and KML formats.

“unknown CMAP subtable format” error when generating PDFs

Whenever I get an unusual exception being thrown by my application that I don’t know how to fix, I Google the error message which generally leads to some kind of explanation of what’s happening. But Google seems to have a hole in its knowledge regarding this “unknown CMAP subtable format” error I was getting. So I’ll try and fill that hole.

Our software uses XSLT to generate XSL:FO files that are then converted to PDF files using AltSoft’s XML2PDF engine. The error was occurring deep inside their assembly and the only clue I could get from looking at the call stack was that the problem was font related. My immediate suspicion was that the error was due to the user using a strange font in their input, but the fonts in use were fairly standard (Arial, Calibri).

I eventually figured out the problem was due to some of the particular characters being used in the input text. It seems the user had pasted some text from Word that contained some characters that weren’t available in the fonts used in the output. Cleaning up the user’s input text stopped the exception being thrown.

Postcode information sites

The release of free postcode data from the Ordnance Survey with very relaxed licensing has led to a raft of sites showing off this data in different ways. I’ve compiled a list of the ones I’ve spotted and will add to this list if I find anymore.

http://www.doogal.co.uk/UKPostcodes.php – Obviously first up is my own site. I’m unable to write an objective review, so won’t.

http://www.free-postcode-maps.co.uk/ – This site doesn’t provide all the postcode data (understandably since it looks like it’s been set up by a company who sell this data in various flavours) but does provide some nice map overlays showing postcode districts and sectors. So nice in fact, I may borrow some of their ideas for my own site.

http://www.britishpostcodes.info/ – I was a bit upset when a search for my own postcode brought up this site above my own site, but it does provide a lot of useful information for each individual postcode. I haven’t figured out where all the data has come from, but once I do, I’ll probably get some of it on my site.

http://postcodeof.co.uk/ – Again a search for a postcode brought up this site above my own, very disappointing. But the site was intriguing, if only for the broken English in the content. A quick domain check showed it was registered by someone in Serbia. I never realised Serbians had a passion for UK postcodes! The other interesting thing is the use of sub-domains for each individual postcode, presumably for SEO reasons. There’s not much data for each postcode, but things may improve.

http://mapit.mysociety.org/ – If you have a postcode and you want to find out what administrative areas it is part of, this site if useful. You can find the electoral constituency, local authority wards, census areas etc and also grab the boundaries of those areas. It also has an API, so you can automate your data grab!

Land Registry to release all data from 1995 onwards

The Land Registry has been releasing its data under the Open Government Licence since the start of 2012. This was great but only covering the last year and a bit meant it wasn’t that useful. But when I popped over to the Land Registry site the other day I noticed they’d got some historical data and are planning to release all of their data from 1995 onwards.

I’ve sucked this data into my site, but I haven’t done anything particularly exciting with it yet. It’s interesting (to me at least) that the average price that I’ve calculated using a straightforward mean (about £230,000) is so different to the average price that the Land Registry comes up with (about £160,000). They do some fancy maths on their numbers but it still seems odd that they are so different.

But me and the Land Registry agree on one thing, the trend. House prices have basically gone nowhere for the last 3 or 4 years.

Postcode electoral constituency data on doogal.co.uk

I’ve just recently imported the latest ONS UK postcode data and added the Westminster constituency data at the same time (except for the Northern Irish postcodes, which still aren’t covered by the OpenData initiative). I’ve only surfaced this on the map pages for individual postcodes, but I’ll show it in other places as time allows. Let me know if you need this chopped up in any particular way.

Website ideas

I often have ideas for websites, but due to a lack of time I never actually get round to doing anything about them. They may well be terrible ideas, or may already exist in some form already, and almost all of them are guaranteed to not make any money, but I present them here in case anyone wants to pick up the baton and run with them.

Random acts of kindness

Say you have some spare cash and you want to give it to a good cause, but you don’t have a good idea about who to give it to. Wouldn’t it be great if there was a website where you could post a message saying ‘I have some money, who wants it?’ and from the other side people could post messages asking for money, help or whatever? When you find someone you want to help, you can contact each other and arrange to hand over the cash (although there needs to be some kind of protection here against receiving shedloads of begging emails).

Domain reminder

So you’re interested in buying that doogal.com domain (oh, that’s just me is it?) but somebody has bought it already and is sitting on it, only being prepared to sell it for a ludicrous 2800 euros? If you’re willing to wait, wouldn’t it be great if there was a website that would send you reminders when the domain name was about to expire? And whenever the status of the domain changed in any other way?

Foreign exchange market

The difference between the official exchange rates and the price you’ll pay to exchange money yourself can be huge. But whenever I convert 100 pounds to euros, there’ll be thousands of other people who want to take their euros and convert them into pounds, so how about a website to match the buyers and sellers? I’ve not really thought through how the exchange takes place, but that’s just a minor detail…

Re-open nominations

This is more than just a website, this is a political movement. Back in the day when I was at university, any election would include RON, re-open nominations. So if you weren’t happy with any of the candidates, you could vote for RON and if enough people voted for him then the election would be rerun. Currently I’m pretty disappointed with all three of our major parties, since there seems to be very little difference between them in terms of policy and they all seem to be led by people who have no experience of the real world. But there is no way to register the fact that I still believe in the democratic process but don’t want to vote for any of the options available to me. So for my take on this idea in real elections, the RON Party would stand in every election, promising to immediately stand down if elected. Along with that, it would lobby the government to include RON as an option in all elections.

Fat cyclists

I enjoy cycling, but have no desire to wear lycra. There are lots of clubs for cyclists, but they are generally full of extremely fit people who cycle 50 miles a day. So how about a web community for people who like to cycle but aren’t necessarily as fit as they could be and don’t take it too seriously so they can get together and have a slow ride together?

Poor man’s XSLT profiling in detail

A couple of years ago I wrote a post giving a vague idea of how to profile XSL transforms without shelling out for Visual Studio Team System. I went back to it recently and realised it didn’t really provide enough information on how to actually perform the profiling. So here’s another go at it.

The first thing we need to do is compile the XSLT into a .NET assembly. For this we need to use the XSLTC.EXE tool that ships with the Windows SDK. You can find this somewhere like this C:\Program Files (x86)\Microsoft SDKs\Windows\v8.0A\bin\NETFX 4.0 Tools. The command-line required will be something like this xsltc.exe /settings:dtd "/out:c:\temp\style.dll" /class:MyXslt "C:\temp\TEST.xsl"

So now we have an assembly for our XSLT file. Next we need a little wrapper program to load it up and call the transform. I used a simple command line app, with code as follows

    static void Main(string[] args)
    {
      XslCompiledTransform transform = new XslCompiledTransform();
      transform.Load(typeof(MyXslt));
      XmlTextReader reader = new XmlTextReader(@"C:\temp\input.xml");
      XmlTextWriter writer = new XmlTextWriter(@"C:\temp\out.html", Encoding.UTF8);
      transform.Transform(reader, writer);
    }

Now we are ready to profile the XSLT. In the past I’ve always recommended AQTime, but my old version of it doesn’t seem to work on Windows 8. I didn’t fancy paying for an upgrade, so looked around for alternatives and found the Eqatec profiler. The free version seems to work pretty well, certainly good enough for my limited needs.

So after running my little test app through the profiler, I had a much better idea of where things were slow in my XSLT. As is often the case, it was due to some XPath using “//element” to find elements. Often you can get away with it, but if the XML document is large or that piece of code is getting hit a lot then it can bite you. 

There is another slight complication with this approach to XSLT profiling. Some of the compiled templates may have a name of “compiler:generated”, meaning it’s pretty tricky to figure out how they relate to the original XSLT. From my experience, it seems that these templates are generated by the compiler itself when it decides to split larger templates into smaller compiled templates. I found ILSpy was pretty useful here to match the compiled code back to the original XSLT.

Traversing JavaScript objects in .NET

I’ve been playing around with hosting a Google Map in a .NET WinForms control, for no good reason other than to see if it can be done. The key part of this integration is calling JavaScript on a web page from .NET. This appears reasonably straightforward, the WebBrowser control has a Document.InvokeScript method that does the trick. If your JavaScript function returns a value, you can pick this up from the return value of InvokeScript.

This works great for simple types, but what if your JavaScript function returns a more complex type? This is where things get a bit more tricky. The returned object has a type of System.__ComObject and initially it looks like this has no useful methods on it. This is the object that is used when calling any COM object, but generally you’ll be able to import a type library to create a .NET friendly wrapper around the raw object. This obviously isn’t the case here.

So my first thought was to find the real underlying type of the __ComObject. This piece of code helped out here. It turns out the actual underlying type was a JScriptTypeInfo but there’s very little information out there about what this type does or how to use it.

But it turns out there’s a much simpler way to access the returned object, cast the returned object as IReflect and use its methods to get properties etc. So the code for my .NET property looks like this

    [Category("Map")]
    public LatLng Centre
    {
      get 
      {
        object centre = webBrowser.Document.InvokeScript("getCentre");
        if (centre == null)
          return new LatLng(0, 0);

        IReflect reflect = centre as IReflect;
        double lat = (double)reflect.InvokeMember("lat", BindingFlags.InvokeMethod, null, 
          centre, null, null, null, null);
        double lng = (double)reflect.InvokeMember("lng", BindingFlags.InvokeMethod, null, 
          centre, null, null, null, null);

        return new LatLng(lat, lng);
      }
      set
      {
        webBrowser.Document.InvokeScript("setCentre", 
          new object[] { value.Latitude, value.Longitude });
      }
    }

Another approach to this would be to have two JavaScript functions, getCentreLat and getCentreLng, which just return simple types, but that could get cumbersome with really complex types.

I’m not sure if it’s possible to pass complex objects into JavaScript but I haven’t needed that yet. Again, for really complex types, passing in each part of them could get messy.

A dog’s life

It’s a hard life for a dog. It all starts when the sun comes up. “I must inform my master that the sun has come up. Woof woof…”.

Some time later “… woof, woof. Ah good, my master is up to see that the sun has come up, I can now go back to sleep”. Thank you dog!

Around the world in 7 years

According to Endomondo, I’ve cycled 2668 miles since I signed up with them, which is just over a tenth of the distance around the world. According to my circle drawing page, this would take me from my house to the North Pole. I’m quite pleased with that.

The distance round the earth is about 24,900 miles, depending on how you measure it. I try to cycle 300 miles a month, so the time it will take me to cycle the whole way round the earth is 24,900/300 which is 83 months, or 6.9 years. So in about 6.1 years I’ll have completed my virtual trip. I’m hoping Endomondo is still around to record my achievement!