Non-geographic postcodes added to UK postcode list

I received a complaint that some postcodes were missing from my UK postcode list that were in the original ONS data. I was a little confused by this, since the import is pretty much automated. But looking at my import code I realised I was ignoring postcodes with no location information. I think initially I’d assumed these were postcodes with duff data so ignored them, but for completeness I’ve now added them to the list, adding a few thousand more postcodes. Since they have no associated location data, there isn’t much interesting information available for each of these postcodes but they may be useful for something… An example.

UK house sales October 2014

I’ve just finished uploading the latest Land Registry UK house sales data to my website. I then realised my annual percentage change calculation was wrong and have fixed that.

UK Postcode data November 2014

I’ve uploaded the latest UK postcode data to my site. There seem to be a few changes to the data (some codes for wards and districts have changed and LSOA data is now available for Scotland). I think I’ve found all the anomalies but let me know if you spot anything odd

More options when calculating route elevations

I’ve added a couple of options to my page for calculating route elevations. You can now select options to avoid highways and toll roads so it should be even more useful!

Goodbye Endomondo, hello Strava and VeloViewer

Back in the day I used to go for bike rides just for the fun of it. Then I discovered Endomondo and was able to analyse every part of a ride. And being a geek, that motivated me to ride more, try to ride faster and try to ride further. I could try to beat my best time for 10 miles or distance covered in an hour. But the number of awards was kind of limited, so my motivation started to flag. I got some more enthusiasm by training for RideLondon 100. But once I’d completed that (or to be more accurate, the 86 miles that I was allowed to ride), I was looking for something else to keep me moving, especially now the days are getting shorter and the weather is deteriorating.
I’d heard of Strava before but I’d discounted it because there was no app for my Windows Phone (what’s new right?). But then I realised Endomondo has the ability to export workouts which can then be imported into Strava. So I gave it a try. And I was immediately hooked. Rather than compare activity over a specific distance/time, I can compare efforts over the same bit of road, a segment. So for any ride I might pass through 20 or 30 segments and I generally beat a personal best on at least one of them. And that helps keep me going.
And for the ultimate geek experience, there’s VeloViewer. This runs on top of Strava and gives me an even more in depth view of my performance on segments. It tells me I have cycled on 1472 segments and I have a score of 86.4 (I have no idea if this is good or bad but it’s headed in an upwards direction which seems like a good thing)
VeloViewer was free but is now asking for £9.99 a year, with a pretty limited free version. This seems like a reasonable amount to pay for something so compelling. Strava also has a premium service but I haven’t seen anything in there that I need. My guess is that Strava will probably start to require premium membership to use their API fully which means any VeloViewer users will need to pay for their Strava account to use VeloViewer. I’d have thought that would be a no-brainer for Strava, since all VeloViewer users are the kind of people who are prepared to pay for a service and are therefore likely to cough up for Strava as well.
Anyway, that should keep me going during the long winter months. Maybe my next motivator will be my own little app running on top of Strava…

House price data now with annual change

The Land Registry sales data on my site now includes the annual price change. I thought this would smooth out the seasonal variations but the annual change is still quite volatile. Even so, it may be of interest.

UK house sales September 2014

The latest house price data from the Land Registry is available on my website and prices continue to defy gravity…

Unable to get property 'setState' of undefined or null reference in ckeditor.js

I’ve been working on the latest and greatest Business Optix product which dynamically creates lots and lots of HTML controls, some of which are CKEditor controls. Whilst testing the app and quickly navigating around it, I started getting an error “Unable to get property 'setState' of undefined or null reference” coming from deep in the bowels of CKEditor. Google astonishingly turned up nothing, but I remembered I’ve had trouble in the past with CKEditor when I didn’t explicitly destroy the editors. The fix was simple, like this

for (name in CKEDITOR.instances) {
  CKEDITOR.instances[name].destroy();
}

Property sales data for August 2014

I’ve uploaded the latest Land Registry data to my website. People comparing the data on my site to news stories like this may be wondering why my data shows a new all time average price, whereas the BBC say prices are yet to reach their previous peak of November 2007. I believe this is due to the Land Registry seasonally adjusting their figures whereas the average values on my site uses a geometric mean of the raw data.

UK property sales data for July 2014

The latest Land Registry data for property sales is now uploaded to my site. Not much can be read into a single month’s figures, but the average sale price has reached an all time high, which I guess pleases a government seeking re-election soon and probably dismays first time buyers.

UK Postcodes August 2014

I have uploaded the ONS postcode dataset for August 2014 to doogal.co.uk. Hopefully everything is in order, let me know if you spot any issues.

Land Registry June 2014 data

I’ve uploaded the Land Registry property sales data for June 2014 to doogal.co.uk. Enjoy!

Setting HttpResponse.StatusDescription silently failing

We recently received a complaint from one of our customers. We provide some fairly simple reporting functionality, that allows more technical users to write their own SQL queries. The customer was building a report and when there was a problem with his SQL, sometimes he’d get a detailed error message, but other times he’d get nothing.

When we try to execute a query and an exception is thrown we catch the exception and write the error message to HttpResponse.StatusDescription so it can be displayed in the browser. This can fail if the message is longer than 512 characters long, but this is fairly obvious since setting the StatusDescription property will throw an exception. That clearly wasn’t the problem here.

I finally managed to reproduce the problem but was still confused about what was causing the issue. Then I spotted the difference between the working case and the non-working case. The non-working case contained new line characters. Thinking about it, this was fairly obviously going to be a problem. The HTTP response would not be valid, since the status description header would be split over two lines. But it would certainly be preferable if .NET threw an exception in this case, rather than just silently failing to set the status description.

So our code now looks something like this

        context.Response.StatusCode = 500;
        context.Response.TrySkipIisCustomErrors = true;
        string description = ex.Message.Replace('\n', ' ').Replace('\r', ' ');
        if (description.Length > 512)
          description = description.Substring(0, 512);

        context.Response.StatusDescription = description;
        context.Response.ContentType = "text/plain";
        context.Response.Write("An error occurred - " + ex.Message);

Land Registry May 2014 data

I’ve uploaded the Land Registry data for May 2014 to doogal.co.uk. For all the talk of a house price bubble, the Land Registry data (arguably the most accurate of all the house price indices) doesn’t seem to show much movement at all over the last few months. Of course it’s a different story in London. There was a story about the housing insanity in Hackney a few months ago, and looking at the sales in East London does show prices shooting up over the last year.

Land Registry house price data for April 2014

The Land Registry house price data for April 2014 is now up on my site

ONS postcode data for May 2014 uploaded to doogal.co.uk

I’ve just uploaded the latest postcode data from the ONS to my site. There are over 2.5 million postcodes in there, alive and dead. My data checks suggest everything is in order, but let me know if you find a problem.

Downloading Javascript generated data

I have a number of web pages that generate some data in text areas using Javascript. The only way users could download this data was to copy and paste the contents of these text areas, but I wanted to add a download button to simplify the process. The problem is that this simply isn’t possible in Javascript. The only client-side solutions I’ve seen either require Flash or are not supported in all browsers.

So I came up with a slightly hacky and inefficient solution. The basic idea is to post the data to the server and get the server to return it to the client as a download. The HTML looks like this

      <form action="download.php" method="post">
        <div>
          <input type="hidden" name="fileName" value="locations.csv" />
          <input type="submit" value="Download" />
        </div>
        <textarea id="geocodedPostcodes" style="width:100%;" rows="20" name="data"></textarea>
      </form>

All that is needed is a hidden field that tells the server-side script what the download file name should be and a text area with a name of “data”.

The server-side script is pretty simple, it looks like this

<?php
  header('Content-Disposition: attachment; filename="' . $_POST["fileName"] . '"');

  // add data
  print($_POST["data"]);
?>

All it does is get the requested file name and echo back the data.

It’s seems a bit crazy (and a waste of bandwidth) that this seems to be the only way to achieve a seemingly simple task, but that looks to be the case. I’d be happy to be proved wrong.

Help me go on a bike ride

Last year I saw the various Ride London rides on the telly and rolling through Kingston and fancied doing it myself. Riding round London and Surrey on traffic-free roads is very appealing, compared to the usual stop-start, take your life in your hands experience of cycling round these parts. So the first chance I had, I applied in the ballot for the Ride London-Surrey ballot. And in January I heard I’d missed out on getting a place.

There was one more option. Sign up with a charity and raise some money and get a guaranteed place. So I decided to try and help out Cancer Research. Why Cancer Research? Primarily because cancer affects so many people at all stages of life but also, on a personal level, one of my partner’s best friends lost her life to cancer a couple of years ago, before she reached the age of 40.

I’ve set up a page for donations, added a link from my website and been amazed by the number of people who don’t know me who’ve already donated. If you’ve found this blog or my website useful, or are just feeling generous, then please consider donating some money. I will certainly appreciate it, as will Cancer Research.

Land Registry March 2014 data uploaded

I’ve uploaded the Land Registry house price data for March 2014 to my website. Now that probably all the sales data for 2013 has come in, it’s plain to see sales volumes were up in 2013 and prices continue to drift upwards

The perils of micro-optimisations

A debate has been raging on my website over the use of StringBuilder.AppendFormat in my exception logger code. OK, raging is something of an exaggeration, there have been two comments in two years. But the point made by two people is that rather than

error.AppendLine("Application: " + Application.ProductName);

I should be using

error.AppendFormat("Application: {0}\n", Application.ProductName);

Since this means I wouldn’t be using string concatenation, which is considered bad for performance reasons. My main reason for not doing anything about this is because I’m lazy, but also because the whole point of this code is that it only runs when an exception is thrown, which hopefully is a pretty rare event, so performance is not a major concern.

But then I wondered what the difference in performance is between these two approaches? So I wrote a little test application that looks like this.

    static void Main(string[] args)
    {
      for (int j = 0; j < 10; j++)
      {
        // try using AppendLine
        Console.WriteLine("AppendLine");
        StringBuilder error = new StringBuilder();
        Stopwatch sw = new Stopwatch();
        sw.Start();
        for (int i = 0; i < 1000000; i++)
        {
          error.AppendLine("Application: " + Application.ProductName);
        }
        sw.Stop();
        Console.WriteLine(sw.ElapsedMilliseconds);

        // try using AppendFormat
        Console.WriteLine("AppendFormat");
        error.Clear();

        sw.Restart();
        for (int i = 0; i < 1000000; i++)
        {
          error.AppendFormat("Application: {0}\n", Application.ProductName);
        }
        sw.Stop();
        Console.WriteLine(sw.ElapsedMilliseconds);
      }

      Console.ReadKey();
    }

The results from this app in milliseconds are as follows (reformatted for clarity)

AppendLine	307	315	321	372	394	370	289	298	300	296
AppendFormat	366	360	362	471	353	359	354	365	365	350

So which is quicker? Well it looks like AppendLine might be marginally quicker. But, much more importantly, who the feck cares? We are repeating each operation 1 million times and the time to execute is still less than half a second. Maybe you can pick holes in my test application, but again I would ask who the feck cares? Either approach is really fast.

And this is the main problem with trying to optimise this kind of stuff. We can spend huge amounts of time figuring out if one approach is quicker than another, but a lot of the time is doesn’t matter. Either the code runs quick enough using any sensible approach, or it’s hit so infrequently that even a really poor implementation will work.

Of course we should consider performance whilst writing code, but we should only use particular approaches when we know they are going to produce more performant code. A good example is the StringBuilder class. We can be pretty sure this is going to be better than using string concatenation, otherwise it wouldn’t exist in the first place. That said, if you’re concatenating two strings I really wouldn’t worry about it.

But the key to writing efficient code is to understand what is slow on a computer. Network operations are slow. Disk access is slow. Because of that, anything that requires large amounts of memory (meaning virtual memory i.e. disk access) is slow. Twiddling bits in memory is quick. Fast code is achieved by avoiding the slow stuff and not worrying about the quick stuff.

And once you’ve written your code and found it doesn’t run ask quick as you’d hoped, don’t jump in and replace calls to AppendLine with calls to AppendFormat, profile your application! Every time I profile an application, I’m always amazed at the causes of the performance bottleneck, it’s rarely where I thought it would be.

If you don’t have a profiler, use poor man’s profiling. There are also free profilers available, I quite liked the Eqatec Profiler which seems to be available from various download sites, although it’s no longer available from Eqatec. But whatever you do, don’t get into Cargo Cult Programming