Land Registry data November 2019

I’ve uploaded the latest property sales data from the Land Registry for England and Wales to my website. Prices continue to creep higher, number of sales are fairly static

Land Registry sales data October 2019

I’ve uploaded the latest property data to my website. Nothing of interest to report…

ONS postcode data for November 2019

I’ve uploaded the latest ONS postcode data to my website. I’ve run my usual sanity checks but let me know if you find any problems

Land Registry data for September 2019

I’ve uploaded the latest house price data to my website. Prices continue on their sluggish ascent, although the annual change is now below 1%. The number of sales seems surprisingly resilient, I guess whatever madness is going on in the UK, people still need to move house

New server

The server running doogal.co.uk was getting a little long in the tooth and with more data getting added every month, was starting to slow down somewhat. So I’ve finally got round to upgrading to a shiny new server with a SSD, more memory and the latest versions of the software used. Hopefully it is all working as expected, but let me know if you spot any problems

Land Registry data August 2019

My server is straining to import the latest house price data, but not to worry, I’m in the process of building a new server with more memory and a faster hard drive which will be up and running shortly.

The latest data is fairly dull, as prices continue to increase very slowly whilst looking like they may head negative some time soon

Land Registry data for July 2019

Another month, another set of house price and sales data from the Land Registry, which I’ve uploaded to my website. Prices are still managing to rise, but at an increasingly slow pace

UK postcode update

I’ve uploaded the latest postcode update from the ONS to my website. I’ve run my usual sanity checks and everything appears correct but let me know if you spot anything amiss.

Land Registry data for June 2019

I’ve uploaded the latest property sales data for England and Wales to my website. Not much to report, the long slow trend towards house price deflation continues

The random address generator is back

My random addresses generator is back. After having my hand slapped by Google Maps, I’ve had to re-implement it using my own database of property sales. Because of that change, the addresses returned are only from England and Wales, but they will now always be actual addresses rather than the occasional A-Road that used to be returned. And you no longer need to provide a Google Maps API key, yay!

I’ve also removed a few other features which I’ll probably start adding back as time permits. Let me know if there’s anything you desperately need.

Land Registry data for May 2019

I’ve uploaded the latest house price data for England and Wales to my website. The data is so boring at the moment I can’t find anything of interest to say about it. The annual rate of inflation has been stuck at 1.7% for the last 3 months.

I will make the prediction that something interesting will happen if/when we leave the EU but since I don’t want to spend my days clearing out abusive messages from the comments section, I won’t say which way I think prices will go…

Isoline routing

Here Maps has an interesting feature that I’ve not seen in other mapping solutions. Called isoline routing, it answers the question “Starting from point A, how far can I travel in X amount of time?”. I’m not sure how useful it is, but it’s certainly interesting to play with, so I’ve added a page implementing it on my site.

Cycling is not supported by the API so I’m using the walking option adjusted for the faster travelling time, although the results may not be completely accurate.

Let me know if there’s anything you’d like to see added to it.

House price data for April 2019

I’ve uploaded the latest Land Registry data for April 2019 to my website. House price inflation continues to drift lower, to just under 2%. It looks like flats are actually dropping in price

ONS postcode data May 2019

I’ve uploaded the latest UK postcode data to my website. As ever if you spot anything amiss, let me know

Help me raise some money for the British Heart Foundation

On June 16th I’ll be riding from London to Brighton to raise funds for the British Heart Foundation. I’ve ridden RideLondon before and was pleasantly surprised at the donations I received from friends, family and complete strangers then so I’m hoping the same happens again.

If you’ve found this blog or my website useful, or are just feeling generous, then please consider donating some money. I will certainly appreciate it, as will the British Heart Foundation.

Why BHF? Well they organise the event so that’s who gets the money. But from a personal perspective, heart disease runs in my family so it’s close to my, er, heart…

Land Registry data for March 2019

I’ve uploaded the latest house price data for England and Wales to my website. Number of sales are fairly static with house price inflation dropping slightly to just under 2%. At the regional level, ignoring the ever volatile central London postcodes (due to a small number of high value transactions), Luton is currently the place with the fastest rising prices and Southall is the place with the fastest dropping prices.

Converting KML maps from Google Maps to Here Maps

When Google went insane and decided to charge excessive amounts for use of their mapping APIs I looked for an alternative. One of the features I needed was support for KML, since my website uses it quite extensively. Which led me to Here Maps.

After a fair amount of work, I managed to convert most of my pages to use Here Maps, but there’s still a few stuck using Google Maps due to features that are unique to Google Maps. This was OK, since my usage was now mostly under the $200 per month of free credit. But recently one of my Google Maps pages started to get a lot of hits due to a new inbound link and I zoomed past $200 free credit into eye-wateringly expensive territory. So time to convert that page to Here Maps.

Converting maps that load KML from Google Maps to Here Maps is generally straightforward, just learn a different API and redo the JavaScript on your page. But the architecture of KML support on the two platforms is different. Google loads the KML on their servers, generates map tiles and uses those to display the KML file. Here Maps loads the KML file in the browser and add markers, polylines etc to the map directly.

The Google approach has one major advantage, it copes well with large KML files. Since the move to Here Maps, I’ve had to stop loading up KML files that I know have more than a few thousand markers in them. Google’s KML support also means you can load up KML files from external sources, whereas Here Maps will generally fail with external KML (unless CORS has been configured to support it on the other server).

Google KML has a few disadvantages. If the KML changes regularly then you’ll probably suffer from caching issues, since the old map tiles can keep getting returned for some time after the KML changes. Also the rendering isn’t as good as Here Maps, since the KML is rendered as an image at each zoom level rather than as live objects on the map

The thing that had stopped me moving this page over to Here Maps was the inability to display remote KML data. Then it struck me that the fix for that was fairly straightforward. Add a local piece of server code that loads up the KML file from the remote source and returns it to the browser so it’s treated as a local URL. That was easy enough to code up. Then I just needed to cope with a few edge cases, Google Maps copes with KMZ files, but Here Maps doesn’t. And some servers didn’t like requests coming from something that wasn’t a browser. So I eventually came up with this

public void ProcessRequest(HttpContext context) {
 context.Response.ContentType = "application/vnd.google-earth.kml+xml";
 var url = context.Request.QueryString["url"];
 var httpRequest = (HttpWebRequest) WebRequest.Create(url);
 httpRequest.Method = "GET";
 // pretend to be a browser
 httpRequest.UserAgent =
  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";

 using(var httpResponse = (HttpWebResponse) httpRequest.GetResponse()) {
  var responseStream = httpResponse.GetResponseStream();

  if (responseStream != null) {
   var archiveStream = new MemoryStream();
   responseStream.CopyTo(archiveStream);
   archiveStream.Position = 0;

   // see if it's a zip file
   try {
    var archive = new ZipArchive(archiveStream);
    using(var stream = archive.Entries[0].Open())
    using(var archiveReader = new StreamReader(stream)) {
     context.Response.Write(archiveReader.ReadToEnd());
    }
   } catch (Exception) {
    archiveStream.Position = 0;
    var reader = new StreamReader(archiveStream);
    var response = reader.ReadToEnd();
    context.Response.Write(response);
    reader.Close();
   }

   // Close both streams.

   responseStream.Close();
  }
 }
}

Updated postcode boundaries

The boundaries for postcode areas, districts and sectors were rather out of date on my website. I originally downloaded them from a website that no longer exists. After searching around the web for updated files, I came up empty handed. The ones I found were either just as out of date as mine or had restrictive licensing attached to them.

So I set about building my own, which are now live on the site.

You can stop reading now unless you have an interest in the technical details of how I built the boundary files.

Since the freely available postcode data doesn’t include postcode unit boundaries, just the centre point of each postcode, the first step was to generate approximate boundaries for postcode units. This can be achieved using a Voronoi diagram, which generates polygons around points so that every location in the polygon is closer to the point than any other points. Although the polygons for each postcode unit are not particularly useful or accurate, combining them together at the district and sector level should give a decent approximation of the actual area covered.

The GIS tool QGIS can calculate Voronoi diagrams so I imported the postcode data into SQL Server and ran the Voronoi tool in QGIS. After several hours it hadn’t produced anything and didn’t give me any indication of when it would complete (it was stuck at 99% complete for most of that time).

So I decided to search for some code that could produce the Voronoi diagram so I could run that and would at least have an idea of where it was in the process. I found this bit of C# code that mostly worked. The output from it was edges rather than polygons, so some further work was required to construct polygons from the edges. Once the polygons were created I put those into SQL Server (adding District and Sector fields for later use) so I could work on them further in QGIS.

The next issue was that the polygons created extended far beyond the shores of the UK. This required the use of the ‘Clip vector by mask layer’ tool in QGIS, using a UK border polygon layer as the mask (note to self for next time I do this – this takes hours to complete). Then the postcode units needed to be merged into districts and sectors. The ‘Dissolve’ tool in QGIS can be used for this, using the District and Sector fields as the dissolve field.

I then had two KML files, one with all postcode districts and one with all postcode sectors. I then wrote some C# code to split the files apart so the individual districts and sectors can be displayed and downloaded.

Land Registry data for February 2019

I’ve just uploaded the latest house price data to my website. Not much to report (2% annual inflation still), but I do have some prettier charts

Land Registry data for January 2019

I’ve uploaded the latest property sales data from the Land Registry to my website. Not much to report, house price inflation remains at 2%, transactions remain flat

UK postcode data for February 2019

I have uploaded the latest UK postcode data from the ONS to my website. I’ve run my usual sanity checks and everything looks OK but let me know if anything appears amiss. Enjoy all the 2,611,698 postcodes (alive and dead)

UK station usage 2018

I’ve added 2018 station usage date for UK stations to my website. Enjoy

Land Registry data for Dec 2018

I’ve uploaded the latest house price data from the Land Registry to my website. Currently every month feels like Groundhog Day, prices continue to rise at about 2%, the number of transactions is stable.

Land Registry house price data November 2018

The good people at the Land Registry were probably taking a well earned vacation, so the data arrived a little later than normal but house price data for England and Wales for November 2018 is now on my site. Not much to report, annual inflation is still sitting at about 2%, the number of transactions are holding steady. Maybe things will get interesting in March, I hear something might be happening around then.