House price data for April 2019

I’ve uploaded the latest Land Registry data for April 2019 to my website. House price inflation continues to drift lower, to just under 2%. It looks like flats are actually dropping in price

ONS postcode data May 2019

I’ve uploaded the latest UK postcode data to my website. As ever if you spot anything amiss, let me know

Help me raise some money for the British Heart Foundation

On June 16th I’ll be riding from London to Brighton to raise funds for the British Heart Foundation. I’ve ridden RideLondon before and was pleasantly surprised at the donations I received from friends, family and complete strangers then so I’m hoping the same happens again.

If you’ve found this blog or my website useful, or are just feeling generous, then please consider donating some money. I will certainly appreciate it, as will the British Heart Foundation.

Why BHF? Well they organise the event so that’s who gets the money. But from a personal perspective, heart disease runs in my family so it’s close to my, er, heart…

Land Registry data for March 2019

I’ve uploaded the latest house price data for England and Wales to my website. Number of sales are fairly static with house price inflation dropping slightly to just under 2%. At the regional level, ignoring the ever volatile central London postcodes (due to a small number of high value transactions), Luton is currently the place with the fastest rising prices and Southall is the place with the fastest dropping prices.

Converting KML maps from Google Maps to Here Maps

When Google went insane and decided to charge excessive amounts for use of their mapping APIs I looked for an alternative. One of the features I needed was support for KML, since my website uses it quite extensively. Which led me to Here Maps.

After a fair amount of work, I managed to convert most of my pages to use Here Maps, but there’s still a few stuck using Google Maps due to features that are unique to Google Maps. This was OK, since my usage was now mostly under the $200 per month of free credit. But recently one of my Google Maps pages started to get a lot of hits due to a new inbound link and I zoomed past $200 free credit into eye-wateringly expensive territory. So time to convert that page to Here Maps.

Converting maps that load KML from Google Maps to Here Maps is generally straightforward, just learn a different API and redo the JavaScript on your page. But the architecture of KML support on the two platforms is different. Google loads the KML on their servers, generates map tiles and uses those to display the KML file. Here Maps loads the KML file in the browser and add markers, polylines etc to the map directly.

The Google approach has one major advantage, it copes well with large KML files. Since the move to Here Maps, I’ve had to stop loading up KML files that I know have more than a few thousand markers in them. Google’s KML support also means you can load up KML files from external sources, whereas Here Maps will generally fail with external KML (unless CORS has been configured to support it on the other server).

Google KML has a few disadvantages. If the KML changes regularly then you’ll probably suffer from caching issues, since the old map tiles can keep getting returned for some time after the KML changes. Also the rendering isn’t as good as Here Maps, since the KML is rendered as an image at each zoom level rather than as live objects on the map

The thing that had stopped me moving this page over to Here Maps was the inability to display remote KML data. Then it struck me that the fix for that was fairly straightforward. Add a local piece of server code that loads up the KML file from the remote source and returns it to the browser so it’s treated as a local URL. That was easy enough to code up. Then I just needed to cope with a few edge cases, Google Maps copes with KMZ files, but Here Maps doesn’t. And some servers didn’t like requests coming from something that wasn’t a browser. So I eventually came up with this

public void ProcessRequest(HttpContext context) {
 context.Response.ContentType = "application/vnd.google-earth.kml+xml";
 var url = context.Request.QueryString["url"];
 var httpRequest = (HttpWebRequest) WebRequest.Create(url);
 httpRequest.Method = "GET";
 // pretend to be a browser
 httpRequest.UserAgent =
  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";

 using(var httpResponse = (HttpWebResponse) httpRequest.GetResponse()) {
  var responseStream = httpResponse.GetResponseStream();

  if (responseStream != null) {
   var archiveStream = new MemoryStream();
   responseStream.CopyTo(archiveStream);
   archiveStream.Position = 0;

   // see if it's a zip file
   try {
    var archive = new ZipArchive(archiveStream);
    using(var stream = archive.Entries[0].Open())
    using(var archiveReader = new StreamReader(stream)) {
     context.Response.Write(archiveReader.ReadToEnd());
    }
   } catch (Exception) {
    archiveStream.Position = 0;
    var reader = new StreamReader(archiveStream);
    var response = reader.ReadToEnd();
    context.Response.Write(response);
    reader.Close();
   }

   // Close both streams.

   responseStream.Close();
  }
 }
}

Updated postcode boundaries

The boundaries for postcode areas, districts and sectors were rather out of date on my website. I originally downloaded them from a website that no longer exists. After searching around the web for updated files, I came up empty handed. The ones I found were either just as out of date as mine or had restrictive licensing attached to them.

So I set about building my own, which are now live on the site.

You can stop reading now unless you have an interest in the technical details of how I built the boundary files.

Since the freely available postcode data doesn’t include postcode unit boundaries, just the centre point of each postcode, the first step was to generate approximate boundaries for postcode units. This can be achieved using a Voronoi diagram, which generates polygons around points so that every location in the polygon is closer to the point than any other points. Although the polygons for each postcode unit are not particularly useful or accurate, combining them together at the district and sector level should give a decent approximation of the actual area covered.

The GIS tool QGIS can calculate Voronoi diagrams so I imported the postcode data into SQL Server and ran the Voronoi tool in QGIS. After several hours it hadn’t produced anything and didn’t give me any indication of when it would complete (it was stuck at 99% complete for most of that time).

So I decided to search for some code that could produce the Voronoi diagram so I could run that and would at least have an idea of where it was in the process. I found this bit of C# code that mostly worked. The output from it was edges rather than polygons, so some further work was required to construct polygons from the edges. Once the polygons were created I put those into SQL Server (adding District and Sector fields for later use) so I could work on them further in QGIS.

The next issue was that the polygons created extended far beyond the shores of the UK. This required the use of the ‘Clip vector by mask layer’ tool in QGIS, using a UK border polygon layer as the mask. Then the postcode units needed to be merged into districts and sectors. The ‘Dissolve’ tool in QGIS can be used for this, using the District and Sector fields as the dissolve field.

I then had two KML files, one with all postcode districts and one with all postcode sectors. I then wrote some C# code to split the files apart so the individual districts and sectors can be displayed and downloaded.

Land Registry data for February 2019

I’ve just uploaded the latest house price data to my website. Not much to report (2% annual inflation still), but I do have some prettier charts