UK postcodes November 2022

I've uploaded the latest postcode date from the ONS to the site. As ever, let me know if you spot anything amiss

Land Registry data for September 2022

I'm in the process of uploading the latest house price data to the website. It looks like house prices are on the way up again, although they are still negative on an annual basis. I would guess that increasing interest rates and the cost of living crisis will likely cause prices to fall in the coming months.

One thing I haven't seen mentioned about increasing mortgage rates is how the effect of them is very different depending on where you are in your mortgage life. For example, compare the increasing payments for someone with a 25 year mortgage compared to someone with 5 years remaining on their £100,000 mortgage

Number of years	Mortgage rate		Percentage increase in payment
	2%	6%
25	£424	£644	52%
5	£1,753	£1,933	10%

The person with 5 years left on their mortgage is mostly paying down the loan whilst the person with 25 years left is mostly paying interest so their payments increase much more. Once again the younger generation are getting walloped, this time by maths rather than their government.

GeoJSON data

I've added GeoJSON versions of some of the KML downloads on the website. If they prove popular, I'll add more as time goes on. Let me know if there is anything you'd particularly like to see.

Land Registry data August 2022

I'm uploading the latest Land Registry data to my website. Sale prices continue to fall but more slowly. And right on cue, the government have reduced Stamp Duty in a bid to get prices moving up again. Unfortunately their other budget changes managed to spook the markets meaning mortgage rates are heading up, so it's quite likely they'll be getting less revenue from Stamp Duty whilst also failing to get an election boost.

Filter segments by Local Legend

A long time ago, one of the users of my Strava Segment Explorer asked for the ability to filter segments by whether they had a Local Legend. Well, now you can! Most segments still don't have the data required so it may not be currently that useful, but over time it should become more helpful. From my own experimentation, it seems the segments that don't have a Local Legend set are ones nobody has ridden recently or that are flagged as hazardous

Downloadable segment lists

I've just added a new feature to the site. When using the segment explorer, you can download a CSV file of the segments you're viewing. The CSV file will include all the segments I know about in the area, which could be more than what is displayed on the map. 

I've limited the maximum area for which the site will return data, mostly to protect my server from malicious or daft users.

The columns returned are currently just what's displayed in the table. If you need other data, let me know. Also let me know if you have any suggestions or find any problems

Land Registry data July 2022

I'm in the process of uploading the latest house sales data for July 2022 to my website. According to the Land Registry, prices are falling on an annual basis but this article caught my eye which claims prices are going up by 10% per year. The Nationwide base their calculations of a different set of data and they probably calculate the numbers differently to me so it's unlikely they would agree completely, but it's still interesting that the direction of travel is completely different. Still, they think the average price of a house is £273,751, my calculations from the Land Registry put it at £266,796, which is in the same ballpark. Maybe they did some funky calculations to remove the effect of the Stamp Duty holiday from last year?

Latest UK postcode data

I've updated my website with the latest postcode data from the ONS. It seems to look sensible but let me know if you spot any problems

GB station usage for 2020-2021

I've updated the GB station usage data for 2020-2021. As you may expect, the usage of stations collapsed during this period of time, something to do with a pandemic, you may remember it.

The data came out last November, but I completely forgot about it. I've now added a reminder for this year and subsequent years.

Land Registry data for June 2022

I'm uploading the latest property data from the Land Registry to my website. Whilst the price of everything else is going through the roof, the price of houses have started going down for the first time in about ten years. The main reason for this is something I hadn't noticed before. Just before the Stamp Duty holiday ended, sales volumes and prices went ballistic. Those figures have now dropped out of the annual price calculations leading to a big drop in prices. The architect of this classic boom and bust could well be our next PM.

Land Registry data for May 2022

I've uploaded the latest property sales data to my website. House price inflation seems to be bouncing around close to zero, which I guess is why the government is talking about 50 year mortgages... 

Land Registry property data for April 2022

I've uploaded the latest property data for England and Wales to my website. The annual figures still show sales falling with prices flat and likely to turn negative next month.

UK postcode data update

I've uploaded the latest UK postcode data to my website. Some people have been asking about the ward boundary changes. These are included in the update but are not complete so I would approach with caution. I made some manual changes so that all postcodes have a ward associated but these will be the old wards in some cases.

From the ONSPD user guide

This release of the NHSPD includes the 2022 ward boundary changes. However, some change orders have yet to be included in OS Boundary-Line including The Cumbria (Structural Changes) Order 2022, The North Yorkshire (Structural Changes) Order 2022 and The Somerset (Structural Changes) Order 2022. Therefore, ward codes E05014171–E05014393 are not yet included. Also, Welsh wards W05001616 –W05001626 in Merthyr Tydfil are yet to be included and Sirhowy ward in Blaenau Gwent should be W05001268 rather than W05000764. In Scotland there is a duplicated ward code – S13003134. This should be S13003134 – Stepps, Chryston and Muirhead in North Lanarkshire and S13002608 – An Taobh Siar agus Nis in Na h-Eileanan Siar

House price for England and Wales March 2022

I'm currently uploading the latest house price data to my website. I really need to improve the performance of the import, it now takes days to complete, but it should be complete some time soon. 

There's a definite trend appearing in the data since last summer, sales are falling and prices have peaked. Annual house price inflation is still positive but it looks like it will go negative very shortly, unless the government intervene again. It's also noticeable that the price drops are mostly being driven by the price of flats, although I have no insight into why that would be the case.

Converting .ashx handlers to .NET 5/6

I've been trying to figure out how to convert my site to .NET 5 or 6 for a long while. The pages are written in PHP (because that was the only thing available to me on my web host at the time) but there are many .ashx handlers that were added when I originally decided to move to ASP.NET but never got round to completing the job. But the .NET Framework isn't going to be improved or added to in the future so moving to .NET 5/6 needs to happen at some point. It's not a simple process, the differences are fairly major, but I think I've finally figured out a path to making the move.

My main concern was that I wanted to have a common code base so I can reuse code and continue to work on my current site whilst I set up the updated one, rather than rewriting the whole thing from scratch. A rewrite would certainly be simpler but the switch over would almost certainly be a disaster.

The first step was to work on assemblies used by the site. The Portability Analyzer helped here in figuring out how portable my code was (although it's not generally too helpful in explaining what to do with APIs that are not available in .NET 5). Fortunately it wasn't too difficult to move .NET Coords to .NET Standard 2, meaning it could be used from old and new .NET. I used the .NET Upgrade Assistant to convert to the latest SDK project format. Luckily there were cross platform versions of the third party assemblies I use.

One other thing I did was rip out Entity Framework, which I've never been able to get along with and move from the Oracle MySql ADO.NET connector to MySqlConnector. Like everything from Oracle on Windows, their MySql connector is a complete dumpster fire.

Once that was done, I decided the next step was to move all my .ashx handler code into a new assembly. So every .ashx file now looks something like

<%@ WebHandler Language="C#" Class="DoogalCode.AdministrativeAreasCSV" %>

I configured that assembly to target .NET 4.8 and .NET 5, since there would have to be differences between them. At this point, the .NET 4.8 assembly compiled fine (although with many warnings about using things from System.Web even though that's perfectly fine in .NET 4.8!) and the .NET 5 assembly failed with a gazillion errors. 

The next step was to build the .NET 5 assembly without errors. I decided the simplest way to do that was to build .NET 5 only versions of all the things the compiler was complaining about. This meant things like HttpContext, HttpRequest, HttpResponse and IHttpHandler. The properties and methods didn't do anything except throw an exception, I didn't want the code to run, just compile. I'd figure out the details of  how to implement those methods as I went through each handler and got them working in .NET 5.

Once I'd done that, I needed a .NET web site. Since my front end is written in PHP, I decided to take a look at PeachPie, which claims to be a PHP compiler for .NET. And I am mightily impressed. I created a new PeachPie project, copied my PHP files across and it just worked. I had my website running in .NET 5 in a few minutes. Admittedly I don't do anything very complicated in my PHP, so I can't be sure it's perfect but it's certainly perfect for my needs.

But I still had a bunch of .ashx handlers to get working in .NET 5. I added a method to my website startup class that let me use my dummy IHttpHandler and HttpContext classes to call into my handlers

private static void MapHandler<THandler>(IApplicationBuilder app, string path) where THandler : IHttpHandler, new()

{

  app.Map(path, (app2) =>

  {

    app2.Run(async context =>

    {

      var handler = new THandler();

      await Task.Run(() =>

      {

        handler.ProcessRequest(new DoogalCode.HttpContext(context));

      });

    });

  });

}

Then for each handler I mapped URL paths to the handler via this method, like so

MapHandler<CountiesCSV>(app, "/CountiesCSV");

Now I could call my handlers and start to fix up the methods in my dummy classes. Which is where I am currently, going through each handler fixing issues as I find them, whilst still being able to work on my live website. Once that is done and has gone live, I may try to figure out how to finally move my PHP code to proper .NET

Addendum - This all worked out really well. I was able to retarget all my classic .NET code to .NET 5, whilst continuing to keep the site running. Switching over to the new website was relatively painless, with little downtime. Moving to .NET 6 was straightforward. I was then able to finally start replacing PHP pages with .NET Razor pages, which I'm currently working through. I've also recently learned that Microsoft are developing their own adapters for System.Web.HttpContext etc to make the transition easier, which I wish I'd known sooner!

House price data for England and Wales February 2022

I've uploaded the latest Land Registry data to my website. I've also added a "Sale Type" column to the data which shows if the sale is a non-standard sale. These include transfers under a power of sale/repossessions, buy-to-lets, transfers to non-private individuals and sales where the property type is classed as ‘Other’. The aggregated data now ignores non-standard sales, although it has not had a major effect.

Land Registry data for Jan 2022

I've uploaded the latest Land Registry data for England and Wales to my website. No analysis, there's more important things to think about.

Distance to the sea for all postcodes

I've added the approximate distance to the sea for every postcode. This is available on the individual postcode pages and as part of the CSV downloads

Distance to a line

Probably only of use to me, but I've added a page that lets you find the shortest distance between a postcode and a straight line

Distance to a line (doogal.co.uk)

UK postcode data update

I've uploaded the latest ONS postcode data to my website. The data has passed my sanity tests, but let me know if you spot anything amiss. The CSV downloads now include a UPRNs column, which is a comma separated list of the UPRNs in the postcode

Land Registry data for December 2021

I've uploaded the latest house price data for England and Wales to my website. And it's as you were, with lower sales and slowing inflation in prices. 

.NET Coords in .NET Core

.NET Coords is now a .NET Standard 2.0 project, meaning it can be used in .NET Framework 4.8 and .NET Core projects

UPRN data

I've uploaded UPRN (Unique Property Reference Number) data for the UK to my website. They are only currently available as a link from viewing a particular postcode (an example) but do let me know if it would be useful to provide them in some other way.

Unfortunately the freely available data doesn't include the thing that would be make this data exceedingly useful, the address. I doubt that is likely to happen since there are many companies whose business is based on selling that data

One other major problem is the open data includes old UPRNs but no indication that the UPRN is no longer active, hopefully this will be addressed in a future release.

There's more information on this data here which covers other issues with it.

C# code to generate a segment FIT file

Someone asked about getting hold of the code to generate FIT files from Strava segments. It's not possible to provide standalone code since it's has dependencies on the rest of my website and a Strava library, but here's what I use which may give people an idea of how to use the rather confusing Garmin FIT API

using System;
using System.Collections.Generic;
using System.IO;
using System.Web;
using Dynastream.Fit;
using Newtonsoft.Json.Linq;
using Strava.Common;
using Strava.Segments;
using Strava.Streams;

public static class FitGenerator
{ 
  private static void AddLeaderEntry(SegmentPointMesg newRecord, int goal, float currentDistance, 
    float totalDistance, byte goalIndex)  
  {
    if (goal != 0)
    {
      var currentGoalTime = (float)Math.Floor(currentDistance / totalDistance * goal);
      newRecord.SetLeaderTime(goalIndex, currentGoalTime);
    }
  }

  private static SegmentLeaderboardEntryMesg GetLeaderboard(int goal, byte goalIndex, 
    SegmentLeaderboardType type, string name)
  {
    var goalLeaderboard = new SegmentLeaderboardEntryMesg();
    if (goal != 0)
    {
      goalLeaderboard.SetMessageIndex(goalIndex);
      goalLeaderboard.SetSegmentTime(goal);
      goalLeaderboard.SetType(type);
      if (!string.IsNullOrEmpty(name))
        goalLeaderboard.SetName(name);
    }

    return goalLeaderboard;
  }

  private static int LatOrLngToSemicircle(double latOrLng)
  {
    return (int)(latOrLng * (Math.Pow(2, 31) / 180));
  }

  private static bool HasPr(List<SegmentStream> pr, Segment seg)
  {
    if (pr != null)
      return true;
    if (seg.AthleteSegmentStats != null && seg.AthleteSegmentStats.PrElapsedTime != null)
      return true;

    return false;
  }

  public static byte[] Generate(Segment seg, int goal, string goalName, int rival, string rivalName, 
    int challenger, string challengerName)
  {
    List<SegmentStream> altitude = null;
    try
    {
      StravaBaseHandler.TryStravaRequest((client) =>
      {
        altitude = client.Streams.GetSegmentStream(seg.Id.ToString(), 
          SegmentStreamType.Altitude | SegmentStreamType.LatLng);
      });
    }
    catch (Exception ex)
    {
      throw new Exception("Failed to get stream for segment " + seg.Id, ex);
    }

    // get all the streams we are interested in
    SegmentStream locationStream = null;
    SegmentStream altitudeStream = null;
    SegmentStream distanceStream = null;

    locationStream = SegmentStream.GetStream(altitude, StreamType.LatLng);
    altitudeStream = SegmentStream.GetStream(altitude, StreamType.Altitude);
    distanceStream = SegmentStream.GetStream(altitude, StreamType.Distance);

    // grab user's best time
    List<SegmentStream> pr = null;
    var athlete = HttpContext.Current.Request.Cookies["athlete"];
    if (athlete != null)
    {
      try
      {
        pr = StravaBaseHandler.GetSegmentPr(seg.Id.ToString(), athlete.Value);
      }
      catch
      {
        // do nothing
      }
    }

    // Generate some FIT messages
    // Every FIT file MUST contain a 'File ID' message as the first message
    var fileIdMesg = new FileIdMesg(); 
    var records = new List<SegmentPointMesg>();

    fileIdMesg.SetType(Dynastream.Fit.File.Segment);
    fileIdMesg.SetManufacturer(Manufacturer.Strava);  // Types defined in the profile are available
    fileIdMesg.SetProduct(65534);
    fileIdMesg.SetTimeCreated(new Dynastream.Fit.DateTime(System.DateTime.Now));
    fileIdMesg.SetSerialNumber(1);
    fileIdMesg.SetNumber(1);

    var fileCreator = new FileCreatorMesg();
    fileCreator.SetHardwareVersion(0);
    fileCreator.SetSoftwareVersion(0);

    // segment ID
    var segmentId = new SegmentIdMesg();
    segmentId.SetName(seg.Name);
    segmentId.SetEnabled(Bool.True);

    if (seg.ActivityType == "Run")
      segmentId.SetSport(Sport.Running);
    else
      segmentId.SetSport(Sport.Cycling);

    byte prIndex = 0;
    byte goalIndex = 1;
    byte rivalIndex = 2;
    byte challengerIndex = 3;
    byte komIndex = 4;
    byte qomIndex = 5;

    if (!HasPr(pr, seg))
    {
      goalIndex--;
      rivalIndex--;
      challengerIndex--;
      komIndex--;
      qomIndex--;
    }

    if (goal == 0)
    {
      rivalIndex--;
      challengerIndex--;
      komIndex--;
      qomIndex--;
    }

    if (rival == 0)
    {
      challengerIndex--;
      komIndex--;
      qomIndex--;
    }

    if (challenger == 0)
    {
      komIndex--;
      qomIndex--;
    }

    if (seg.KOM() == 0)
    {
      qomIndex--;
    }

    segmentId.SetSelectionType(SegmentSelectionType.Starred);
    segmentId.SetUuid(Guid.NewGuid().ToByteArray());
    if (HasPr(pr, seg))
      segmentId.SetDefaultRaceLeader(prIndex);

    if (seg.Map.Polyline == "")
      throw new Exception("The segment has no map data (which is somewhat unexpected), " + 
        "I can't generate the FIT file");

    var points = PolylineDecoder.Decode(seg.Map.Polyline);

    // figure out SW and NE
    double swLat = 1000;
    double swLong = 1000;
    double neLat = -1000;
    double neLong = -1000;
    foreach (var pt in points)
    {
      swLat = Math.Min(swLat, pt.Latitude);
      swLong = Math.Min(swLong, pt.Longitude);
      neLat = Math.Max(neLat, pt.Latitude);
      neLong = Math.Max(neLong, pt.Longitude);
    }

    // segment info
    var lap = new SegmentLapMesg();
    lap.SetUuid(Guid.NewGuid().ToByteArray());
    lap.SetTotalDistance(seg.Distance);
    lap.SetTotalAscent((ushort)seg.TotalElevationGain);
    lap.SetSwcLat(LatOrLngToSemicircle(swLat));
    lap.SetSwcLong(LatOrLngToSemicircle(swLong));
    lap.SetNecLat(LatOrLngToSemicircle(neLat));
    lap.SetNecLong(LatOrLngToSemicircle(neLong));
    lap.SetMessageIndex(1);
    lap.SetStartPositionLat(LatOrLngToSemicircle(points[0].Latitude));
    lap.SetStartPositionLong(LatOrLngToSemicircle(points[0].Longitude));
    lap.SetEndPositionLat(LatOrLngToSemicircle(points[points.Count - 1].Latitude));
    lap.SetEndPositionLong(LatOrLngToSemicircle(points[points.Count - 1].Longitude));

    // goal entry
    var goalLeaderboard = GetLeaderboard(goal, goalIndex, SegmentLeaderboardType.Goal, goalName);
    var rivalLeaderboard = GetLeaderboard(rival, rivalIndex, SegmentLeaderboardType.Rival, rivalName);
    var challengerLeaderboard = 
      GetLeaderboard(challenger, challengerIndex, SegmentLeaderboardType.Challenger, challengerName);
    var komLeaderboard = GetLeaderboard(seg.KOM(), komIndex, SegmentLeaderboardType.Kom, "");
    var qomLeaderboard = GetLeaderboard(seg.QOM(), qomIndex, SegmentLeaderboardType.Qom, "");

    SegmentStream prDistanceStream = null;
    SegmentStream prTimeStream = null;
    if (pr != null)
    {
      prDistanceStream = SegmentStream.GetStream(pr, StreamType.Distance);
      prTimeStream = SegmentStream.GetStream(pr, StreamType.Time);
    }

    SegmentLeaderboardEntryMesg prLeaderboard = null;
    if (HasPr(pr, seg))
    {
      prLeaderboard = new SegmentLeaderboardEntryMesg();
      prLeaderboard.SetMessageIndex(prIndex);
      var prTime = seg.AthleteSegmentStats.PrElapsedTime;
      prLeaderboard.SetSegmentTime(prTime);
      prLeaderboard.SetType(SegmentLeaderboardType.Pr);
    }

    for (var i = 0; i < locationStream.Data.Count; i++)
    {
      var newRecord = new SegmentPointMesg();
      var latLng = locationStream.Data[i];
      if (latLng != null)
      {
        var intLat = LatOrLngToSemicircle(Convert.ToDouble(((JArray)(latLng)).First));
        newRecord.SetPositionLat(intLat);
        var intLng = LatOrLngToSemicircle(Convert.ToDouble(((JArray)(latLng)).Last));
        newRecord.SetPositionLong(intLng);
      }

      if (altitudeStream != null)
        newRecord.SetAltitude(Convert.ToSingle(altitudeStream.Data[i]));
      var currentDistance = Convert.ToSingle(distanceStream.Data[i]);
      newRecord.SetDistance(currentDistance);
      newRecord.SetMessageIndex((ushort)i);

      if (HasPr(pr, seg))
      {
        if (pr != null)
        {
          // add PR time
          // find index of the PR item that is at or beyond current distance
          var pos = prDistanceStream.Data.Count - 1;
          for (var j = 0; j < prDistanceStream.Data.Count; j++)
          {
            if (Convert.ToSingle(prDistanceStream.Data[j]) - 
              Convert.ToSingle(prDistanceStream.Data[0]) >= currentDistance)
            {
              pos = j;
              break;
            }
          }
          // find time at that distance
          if (pos > 0)
          {
            var prStartDistance = Convert.ToSingle(prDistanceStream.Data[pos - 1]) - 
              Convert.ToSingle(prDistanceStream.Data[0]);
            var prEndDistance = Convert.ToSingle(prDistanceStream.Data[pos]) - 
              Convert.ToSingle(prDistanceStream.Data[0]);
            var fraction = (currentDistance - prStartDistance) / (prEndDistance - prStartDistance);

            var prTime = Convert.ToSingle(prTimeStream.Data[pos - 1]) + fraction * 
              (Convert.ToSingle(prTimeStream.Data[pos]) - Convert.ToSingle(prTimeStream.Data[pos - 1]));

            newRecord.SetLeaderTime(prIndex, 
              (float)Math.Floor(prTime - Convert.ToSingle(prTimeStream.Data[0])));
          }
          else
          {
            newRecord.SetLeaderTime(prIndex, 0);
          }
        }
        else
        {
          AddLeaderEntry(newRecord, seg.AthleteSegmentStats.PrElapsedTime.Value, currentDistance, 
            seg.Distance, prIndex);
        }
      }

      // add goal time
      AddLeaderEntry(newRecord, goal, currentDistance, seg.Distance, goalIndex);
      AddLeaderEntry(newRecord, rival, currentDistance, seg.Distance, rivalIndex);
      AddLeaderEntry(newRecord, challenger, currentDistance, seg.Distance, challengerIndex);
      AddLeaderEntry(newRecord, seg.KOM(), currentDistance, seg.Distance, komIndex);
      AddLeaderEntry(newRecord, seg.QOM(), currentDistance, seg.Distance, qomIndex);

      records.Add(newRecord);
    }

    // Create file encode object
    var encodeDemo = new Encode(ProtocolVersion.V20);

    using (var fitDest = new MemoryStream())
    {

      // Write our header
      encodeDemo.Open(fitDest);

      // Encode each message, a definition message is automatically generated and output if necessary
      encodeDemo.Write(fileIdMesg);
      encodeDemo.Write(fileCreator);
      encodeDemo.Write(segmentId);
      if (prLeaderboard != null)
        encodeDemo.Write(prLeaderboard);
      if (goal != 0)
        encodeDemo.Write(goalLeaderboard);
      if (rival != 0)
        encodeDemo.Write(rivalLeaderboard);
      if (challenger != 0)
        encodeDemo.Write(challengerLeaderboard);
      if (seg.KOM() != 0)
        encodeDemo.Write(komLeaderboard);
      if (seg.QOM() != 0)
        encodeDemo.Write(qomLeaderboard);

      encodeDemo.Write(lap);
      encodeDemo.Write(records);
      encodeDemo.Close();

      fitDest.Flush();
      fitDest.Position = 0;
      return fitDest.ToArray();
    }
  }
}

Land Registry November 2021

My server is churning through the latest Land Registry data (for November 2021) and the top level figures are now available on my website. Prices continue upwards although at a slower pace, the number of sales are still subdued after the end of the Stamp Duty holiday

Update to .NET Coordinates

I've released a new version of .NET Coordinates to add support for the Irish Transverse Mercator coordinate system. This was introduced 20 years ago to replace the Irish National Grid and I only heard of it about a week ago... I've also made some changes under the hood to make adding new coordinate systems easier in future so if your favourite isn't available, let me know.

Distance to sea improvements

I've been showing some love to the Distance to Sea page. I've improved the precision of the coastline I'm using for the calculation, although it's still not super accurate since it would be really slow to load. I've also improved the algorithm used to calculate the distance so it should give more sensible results. I've also made a few other minor tweaks that should make it easier to use.

Land Registry data for October 2021

I've uploaded the latest Land Registry data to my website. It's been a few months since the end of the Stamp Duty holiday and it's now clear the number of sales has fallen quite dramatically since then. Prices are still increasing but at a slower rate than previously.

UK postcode data update

I've uploaded the November 2021 ONS postcode data to the site. It all looks good but let me know of any problems

Constituency MPs

I've updated my constituency data to include the MP and their party. I've also improved the KML to use markers representing each party

Bus stops

Everybody loves buses, right? So everybody must also love bus stops. A while ago someone asked about adding the nearest bus stop to the postcode data alongside the nearest train station and I've since imported the NapTan bus stop data into my database. Nearest bus stops can now be seen when viewing individual postcodes, train stations and football stadiums.

What I haven't done is add the nearest bus stop to the postcodes data. I'm wondering if that's actually a useful thing to include since a bus stop a little further away may have more regular services or the buses may go in a more useful direction. But if anyone does want this data or wants other bus stop related data, let me know

Land Registry data for September 2021

I've uploaded the latest house price data from the Land Registry to my website. Prices are still rising although the rate of increase seems to have peaked

Welsh counties

I've added Welsh preserved counties to the counties data. If anybody knows of any KML files of the boundaries of these counties, I'd be interested in reusing the data

Find postcodes improvement

A minor update to the find postcodes page, you can now enter a latitude/longitude rather than clicking on the map

Find postcodes (doogal.co.uk) 

Land Registry data for Aug 2021

A bit later than usual this month due to packing my daughter off to university, but the Land Registry data for August is now being uploaded to my website. Covid and the end of the Stamp Duty holiday continue to make it difficult to draw any conclusions from the data

NUTS / ITL data now available

NUTS is yet another way of dividing countries into smaller subdivisions. It's been renamed to ITL since Brexit, although the defined areas currently remain the same.

I've added level 2 and level 3 data to the main postcode CSV downloads (NUTS level 1 data was already available as the Region column). If you'd like it available in any other formats, let me know.

UK Postcodes (doogal.co.uk)

Loading and saving shapes

I've made a little improvement to the shape generator page so you can load and save multiple collections of shapes. That means I can now use it for what I want to use it for!

Map shapes generator (doogal.co.uk)

Updated built up areas CSV download

The CSV download for built up areas now includes population, households and county data

Built up areas (doogal.co.uk)

Population and households data is still from 2011 unfortunately, 2021 census data has not been released yet

Updated map shapes generator

I've updated the polyline generator to include support for adding markers

Map shapes generator (doogal.co.uk) 

The cycle lane to hell

It started with me noticing I wasn't far from the top of the leaderboard for number of roads covered in Claygate on Wandrer. Then came the realisation that the only way to top that leaderboard was to ride along some of the A3. A bit of a search of Google Maps showed what appeared to be a cycle lane

So somebody at some point thought it was a sensible idea to ride on the A3 at the Esher junction. And to be fair, it didn't look too bad, other than the laughably narrow cycle lane on the slip road. Although the traffic is fairly fast moving at that point, there's a hard shoulder, which I'm guessing is where cyclists who want to live are meant to ride. But the hard shoulder disappears when the road merges with the Kingston bypass and you can't exit onto that road. You also can't exit at the next junction, by which point the A3 has turned into a nasty, busy, suburban thoroughfare. It's only after passing that junction and avoiding the traffic coming on via the slip road that you can actually exit onto a local road. 

I've searched up and down the road on Street View trying to find the officially sanctioned route for cyclists but I can find no further signs for cyclists. Did they just forget to finish this bit of infrastructure or was the planner actively trying to kill people?

I did figure out a couple of possible ways to escape the A3 before it gets too unpleasant. There's a tunnel under the A3 with a path into Claygate. That does involve negotiating a barbed wire fence but is otherwise OK. 

The two times I've been down that path the tunnel has been flooded so you may not have much luck if you need to get through it.

The other option would be to hop over onto the Kingston bypass via what looks like an unofficial junction drivers have created when they've realised they can't get there via a road. That involves negotiating the cars merging from the Kingston bypass and then probably stopping to get onto the grass, but probably OK if there's not much traffic

Find postcodes improvement

I've made a small modification to the Find Postcodes page so you can now upload a KML file and see the shapes within it rather than having to upload and draw your own shape around it. Let me know if there any problems with it

Latest Land Registry data

I'm in the process of uploading the latest Land Registry data to my site for July 2021. Although the sales data won't be complete yet, the effect of the end of the Stamp Duty holiday for higher value properties already appears to be visible in sale prices, with a noticable increase in prices in June and a big reduction in July

Latest UK postcode data

I've just uploaded the latest ONS postcode data to my website. I've run my usual sanity checks, but let me know if something looks wrong

Strava route viewer now using new maps

The Strava route viewer is now using the same maps as the segment explorer. Again hopefully I've not broken anything other than IE support

Strava segment explorer with new maps

I’ve updated my Strava segment explorer page to use different maps. These maps include more tracks and paths and also show contour lines to give you more of an idea of the gradient you’ll be facing. Other than that, it should work just the same as before. Of course I may have introduced a bug or two so let me know if you have any problems. Unfortunately I can no longer support Internet Explorer with these maps.

Finding roads in a polygon

I’ve made some improvements to the Find Places page. Support for roads is now better since it displays the actual path of the road, rather than just the centre point. So if you question is ‘what are the roads in this polygon?’, you can now find the answer.

Now also available on Twitter

I’ve decided to start posting updates on Twitter for people who prefer that for notifications

Follow me here

Land Registry data for June 2021

I’ve uploaded the latest Land Registry to my website. Prices continue on their upward trend with the number of sales looking to have almost returned to their pre-pandemic level. Meanwhile, economists suggest the Stamp Duty holiday was a bad idea, with any savings being wiped out by price increases.

Property sales May 2021

I’ve uploaded the latest Land Registry data to my website. Transactions continue to fall, prices continue to rise…

Land Registry sales data for April 2021

Another month, another drop of property data. And another month showing a big increase in property prices. You can find the data here. Whether intentional or not, the Stamp Duty holiday has caused a house price boom.

UK postcode data May 2021

I’ve uploaded the latest postcode data from the ONS to my website. I’ve run the usual sanity checks but as always let me know if you spot anything amiss

Land Registry data for March 2021

I’ve uploaded the latest house price data from the Land Registry to my website. It seems the Stamp Duty holiday, put in place to save the housing market during the pandemic, has actually caused something of a house price boom. Time will tell what happens at the end of the holiday.

House price data for February 2021

I’ve uploaded the latest house price data for England and Wales from the Land Registry to my website. We’re still in the middle of a pandemic but house prices continue to defy gravity, almost certainly due to the stamp duty holiday. That is due to end in June so we’ll see what happens after that

Land registry sales data for January 2021

I’ve uploaded the latest data from the Land Registry to my website. House price inflation seems to be increasing and the number of sales seems to be heading back to pre-pandemic levels.

Updated UK postcode data

I’ve updated the UK postcode data on my website to use the February 2021 ONS release. It’s passed my usual sanity checks but if you see anything amiss, let me know.

Land Registry property data for December 2020

I’ve uploaded the latest property data for England and Wales to my website. I’ve also made some changes to the charts so they hopefully give a clearer picture of trends in sales and prices. I think the most striking thing I see is that the number of sales over the last year have dropped to levels last seen in the days after the global financial crash. But unlike those days, this hasn’t had the same effect on prices which have continued to rise. I guess that’s down to the removal of stamp duty, so it will be interesting to see what happens when that ends.

Land Registry data for November 2020

I’ve uploaded the latest house sales data from the Land Registry to my website.

Website updates

I always forget to announce when new features have been added to the site, so here’s a few things that have been added over the last few months

• Find places – A new page to find places and other points of interest in the UK (driven by the OS Open Names dataset)
• Added the latest usage data for British stations
• Added football stadiums in Northern Ireland

If you find any issues, let me know

Land Registry property sales data October 2020

I’ve uploaded the October 2020 Land Registry property sales data for England and Wales to my website. I really can’t think of anything to say about it!

Postcode data for November 2020

I’ve uploaded the latest UK postcode data from the ONS to my website. I have checked that it looks OK but I often miss something so let me know if you spot anything amiss

England and Wales house price data update

The latest update from the Land Registry for September 2020 is now live on my site. Data still seems to be slow in coming through so the average sale price and number of sales is still volatile. I’m guessing the numbers from the summer are now complete which show prices flatlining and sales volumes much lower than normal

Land Registry data for August 2020

The latest property data for England and Wales is now on my website. The number of sales has certainly fallen over the last few months, even though I’ve been hearing of a property boom. And I’m not seeing much evidence of prices rising much either

Land Registry data for July 2020

The latest house price data from the Land Registry is now on my website. Sales are still way down but it’s too early to see what affect the Stamp Duty holiday has had, I’d imagine sales will be up over the coming months

UK postcode data for August 2020

I’ve uploaded the latest postcode data from the ONS to my website. I’ve run my usual sanity checks and it all looks OK, but do let me know if you spot any anomalies

House price data June 2020

I’ve uploaded the Land Registry data for June 2020 to my website. I reported a couple of months ago that there had been 7 sales in April. This has now been revised up to over 20,000 sales. Whilst not as terrible as the initial figures suggested, that’s still much lower activity than we’ve been seeing over the last few years. And although there’s been a marked drop in the prices paid, those prices haven’t affected the annual averages yet.

Property sales May 2020

I’ve uploaded the latest Land Registry property data to my site. There’s a continued lack of sales being completed, although last month’s figures have been revised upwards from 7 to 279 sales. That is still dramatically lower than the usual number of sales

If you’re wondering why the current average sale price has dropped significantly but the annual change in price has increased, this is due to the tiny number of sales happening, the sales dropping out of the other end are making a much bigger difference to the stats.

UK postcode data for May 2020

The ONS have published their latest postcode data which is now uploaded to my site. As ever, I’ve run some sanity checks (codes for administrative areas have changed yet again!) and everything looks to be correct. Let me know if you see anything amiss

(lack of) property sales April 2020

What happens to property sales in the middle of a pandemic? Perhaps unsurprisingly they fall off a cliff. According to the Land Registry, there were 7 sales in England and Wales in April, although that will probably go up a little as the data comes in for future months. With such a small number of sales, the price statistics are fairly meaningless

Land Registry data for March 2020

In the before days, people bought and sold houses. I’ve uploaded the data for England and Wales from March to my website. Since the data is never complete for the previous month, it’s difficult to see at this point the effect of the lockdown, but I assume it will become more obvious next month.

Land Registry data for February 2020

It may feel like a lifetime ago, but in the not too distant past people were able to buy and sell houses. I’ve uploaded the latest Land Registry data to my website which is fairly uninteresting, with prices still ticking up slowly. I imagine next month’s data will look very different.

Property data for January 2020

I’ve uploaded the latest Land Registry data for England and Wales to my website, covering sales up to January 2020. Prices continue to creep up, number of sales are hovering around the level they’ve been at since 2013 (when Help To Buy was introduced)

UK postcode data for February 2020

I’ve uploaded the latest postcode data from the ONS to my website. I’ve run the usual sanity checks but do let me know if you spot anything amiss

Moving from a Garmin 810 to a Garmin 830

About 5 years ago, I decided I needed to buy myself a dedicated bike computer, rather than the phone I’d been using up to that point. Primarily this was so I could ride for further since my phone generally died after a couple of hours of constant screen and GPS action. I went for the Garmin 810.

In a sense the 810 was a step back from a phone with its tiny screen and washed out colours, but I grew to love its turn by turn navigation, even if it was also the cause of my major annoyance with it, the occasional crash that only seemed to happen when using the turn by turn navigation.

But recently the USB socket packed in, my computer no longer recognised the Garmin when I plugged it in. Although I could still download ride data via my phone’s Bluetooth connection, I now had no way of uploading new routes to the device, so decided to treat myself with a 830. Those crashes had put me off Garmin devices somewhat but when I compared what was available from other companies, I came to the conclusion the 830 still looked like the best choice.

So this is a mini review of how the world of Garmin has moved on in the last 5 years and whether an upgrade is worth it.

On first sight, the 830 looks fairly similar to the 810 with almost exactly the same dimensions. The buttons have moved from the top to the front of the device which makes it rather difficult to press them if the device is mounted on your stem. Because of this, I’ve switched to using the out front bracket which is thankfully included with the device.

It’s when you turn it on, you start to notice the differences. The screen has a much higher resolution and the colours are much improved over the 810. The software should be familiar to anyone who has used an 810, but there have been many small tweaks and changes. Moving around the screens is very responsive and I’ve had no problems when wearing gloves.

I remember when I bought the 810, only a very basic base map was provided. I could buy a proper map from Garmin or try to figure out how to download OSM maps and install them. I went for OSM maps initially but eventually bought Garmin maps in the vain hope it would fix the crashes I was experiencing. Now the 830 comes with its own maps

Integration with my phone has moved on since the 810. I can now get routes directly on to the Garmin via the Strava app, rather than having to download from Strava and upload to the bike computer. I even get notifications from my phone appearing on the Garmin. Admittedly I’m generally on my bike to get away from these kind of things, so this may not be something I leave enabled, but I can imagine it would be useful when trying to meet up with someone.

I was always a big fan of turn by turn navigation. This has been improved so that when I go off route, the 830 will recalculate the route, even if this often just means telling me to make a U-turn. I even get told about sharp corners ahead. it can get a bit confused if a route covers the same bit of road more than once, trying to send me the wrong way, but it figures it out fairly quickly.

I’ve had the 830 for just over a month and ridden just over 300 miles. In that time I’ve had zero crashes. I wouldn’t like to say the software reliability problems of the past are gone (since crashes seemed to happen more frequently on longer rides which I’ve not really been doing due to the winter months) but I’m feeling more confident than I did with the 810.

I think the only complaint I have about the 830 is a problem I have with most bike computers, the price. I have a Garmin sat-nav for our car which provides pretty much the same functionality as the 830, but for a quarter of the price. There are differences in the software and the 830 integrates with heart rate monitors and power meters, but can such a vast difference in price be justified? I guess if this keeps going for five years, then maybe it can. Either way, I’ve been very impressed after my first month

Land Registry data December 2019

I’ve uploaded the latest property sales data for England and Wales to my website. Nothing of note to report, annual house price inflation is stuck at 1%, sales seem to be holding steady.

Land Registry data November 2019

I’ve uploaded the latest property sales data from the Land Registry for England and Wales to my website. Prices continue to creep higher, number of sales are fairly static

Land Registry sales data October 2019

I’ve uploaded the latest property data to my website. Nothing of interest to report…

ONS postcode data for November 2019

I’ve uploaded the latest ONS postcode data to my website. I’ve run my usual sanity checks but let me know if you find any problems

Land Registry data for September 2019

I’ve uploaded the latest house price data to my website. Prices continue on their sluggish ascent, although the annual change is now below 1%. The number of sales seems surprisingly resilient, I guess whatever madness is going on in the UK, people still need to move house

New server

The server running doogal.co.uk was getting a little long in the tooth and with more data getting added every month, was starting to slow down somewhat. So I’ve finally got round to upgrading to a shiny new server with a SSD, more memory and the latest versions of the software used. Hopefully it is all working as expected, but let me know if you spot any problems

Land Registry data August 2019

My server is straining to import the latest house price data, but not to worry, I’m in the process of building a new server with more memory and a faster hard drive which will be up and running shortly.

The latest data is fairly dull, as prices continue to increase very slowly whilst looking like they may head negative some time soon

Land Registry data for July 2019

Another month, another set of house price and sales data from the Land Registry, which I’ve uploaded to my website. Prices are still managing to rise, but at an increasingly slow pace

UK postcode update

I’ve uploaded the latest postcode update from the ONS to my website. I’ve run my usual sanity checks and everything appears correct but let me know if you spot anything amiss.

Land Registry data for June 2019

I’ve uploaded the latest property sales data for England and Wales to my website. Not much to report, the long slow trend towards house price deflation continues

The random address generator is back

My random addresses generator is back. After having my hand slapped by Google Maps, I’ve had to re-implement it using my own database of property sales. Because of that change, the addresses returned are only from England and Wales, but they will now always be actual addresses rather than the occasional A-Road that used to be returned. And you no longer need to provide a Google Maps API key, yay!

I’ve also removed a few other features which I’ll probably start adding back as time permits. Let me know if there’s anything you desperately need.

Land Registry data for May 2019

I’ve uploaded the latest house price data for England and Wales to my website. The data is so boring at the moment I can’t find anything of interest to say about it. The annual rate of inflation has been stuck at 1.7% for the last 3 months.

I will make the prediction that something interesting will happen if/when we leave the EU but since I don’t want to spend my days clearing out abusive messages from the comments section, I won’t say which way I think prices will go…

Isoline routing

Here Maps has an interesting feature that I’ve not seen in other mapping solutions. Called isoline routing, it answers the question “Starting from point A, how far can I travel in X amount of time?”. I’m not sure how useful it is, but it’s certainly interesting to play with, so I’ve added a page implementing it on my site.

Cycling is not supported by the API so I’m using the walking option adjusted for the faster travelling time, although the results may not be completely accurate.

Let me know if there’s anything you’d like to see added to it.

House price data for April 2019

I’ve uploaded the latest Land Registry data for April 2019 to my website. House price inflation continues to drift lower, to just under 2%. It looks like flats are actually dropping in price

ONS postcode data May 2019

I’ve uploaded the latest UK postcode data to my website. As ever if you spot anything amiss, let me know

Help me raise some money for the British Heart Foundation

On June 16th I’ll be riding from London to Brighton to raise funds for the British Heart Foundation. I’ve ridden RideLondon before and was pleasantly surprised at the donations I received from friends, family and complete strangers then so I’m hoping the same happens again.

If you’ve found this blog or my website useful, or are just feeling generous, then please consider donating some money. I will certainly appreciate it, as will the British Heart Foundation.

Why BHF? Well they organise the event so that’s who gets the money. But from a personal perspective, heart disease runs in my family so it’s close to my, er, heart…

Land Registry data for March 2019

I’ve uploaded the latest house price data for England and Wales to my website. Number of sales are fairly static with house price inflation dropping slightly to just under 2%. At the regional level, ignoring the ever volatile central London postcodes (due to a small number of high value transactions), Luton is currently the place with the fastest rising prices and Southall is the place with the fastest dropping prices.

Converting KML maps from Google Maps to Here Maps

When Google went insane and decided to charge excessive amounts for use of their mapping APIs I looked for an alternative. One of the features I needed was support for KML, since my website uses it quite extensively. Which led me to Here Maps.

After a fair amount of work, I managed to convert most of my pages to use Here Maps, but there’s still a few stuck using Google Maps due to features that are unique to Google Maps. This was OK, since my usage was now mostly under the $200 per month of free credit. But recently one of my Google Maps pages started to get a lot of hits due to a new inbound link and I zoomed past $200 free credit into eye-wateringly expensive territory. So time to convert that page to Here Maps.

Converting maps that load KML from Google Maps to Here Maps is generally straightforward, just learn a different API and redo the JavaScript on your page. But the architecture of KML support on the two platforms is different. Google loads the KML on their servers, generates map tiles and uses those to display the KML file. Here Maps loads the KML file in the browser and add markers, polylines etc to the map directly.

The Google approach has one major advantage, it copes well with large KML files. Since the move to Here Maps, I’ve had to stop loading up KML files that I know have more than a few thousand markers in them. Google’s KML support also means you can load up KML files from external sources, whereas Here Maps will generally fail with external KML (unless CORS has been configured to support it on the other server).

Google KML has a few disadvantages. If the KML changes regularly then you’ll probably suffer from caching issues, since the old map tiles can keep getting returned for some time after the KML changes. Also the rendering isn’t as good as Here Maps, since the KML is rendered as an image at each zoom level rather than as live objects on the map

The thing that had stopped me moving this page over to Here Maps was the inability to display remote KML data. Then it struck me that the fix for that was fairly straightforward. Add a local piece of server code that loads up the KML file from the remote source and returns it to the browser so it’s treated as a local URL. That was easy enough to code up. Then I just needed to cope with a few edge cases, Google Maps copes with KMZ files, but Here Maps doesn’t. And some servers didn’t like requests coming from something that wasn’t a browser. So I eventually came up with this

public void ProcessRequest(HttpContext context) {
 context.Response.ContentType = "application/vnd.google-earth.kml+xml";
 var url = context.Request.QueryString["url"];
 var httpRequest = (HttpWebRequest) WebRequest.Create(url);
 httpRequest.Method = "GET";
 // pretend to be a browser
 httpRequest.UserAgent =
  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";

 using(var httpResponse = (HttpWebResponse) httpRequest.GetResponse()) {
  var responseStream = httpResponse.GetResponseStream();

  if (responseStream != null) {
   var archiveStream = new MemoryStream();
   responseStream.CopyTo(archiveStream);
   archiveStream.Position = 0;

   // see if it's a zip file
   try {
    var archive = new ZipArchive(archiveStream);
    using(var stream = archive.Entries[0].Open())
    using(var archiveReader = new StreamReader(stream)) {
     context.Response.Write(archiveReader.ReadToEnd());
    }
   } catch (Exception) {
    archiveStream.Position = 0;
    var reader = new StreamReader(archiveStream);
    var response = reader.ReadToEnd();
    context.Response.Write(response);
    reader.Close();
   }

   // Close both streams.

   responseStream.Close();
  }
 }
}

Updated postcode boundaries

The boundaries for postcode areas, districts and sectors were rather out of date on my website. I originally downloaded them from a website that no longer exists. After searching around the web for updated files, I came up empty handed. The ones I found were either just as out of date as mine or had restrictive licensing attached to them.

So I set about building my own, which are now live on the site.

You can stop reading now unless you have an interest in the technical details of how I built the boundary files.

Since the freely available postcode data doesn’t include postcode unit boundaries, just the centre point of each postcode, the first step was to generate approximate boundaries for postcode units. This can be achieved using a Voronoi diagram, which generates polygons around points so that every location in the polygon is closer to the point than any other points. Although the polygons for each postcode unit are not particularly useful or accurate, combining them together at the district and sector level should give a decent approximation of the actual area covered.

The GIS tool QGIS can calculate Voronoi diagrams so I imported the postcode data into SQL Server and ran the Voronoi tool in QGIS. After several hours it hadn’t produced anything and didn’t give me any indication of when it would complete (it was stuck at 99% complete for most of that time).

So I decided to search for some code that could produce the Voronoi diagram so I could run that and would at least have an idea of where it was in the process. I found this bit of C# code that mostly worked. The output from it was edges rather than polygons, so some further work was required to construct polygons from the edges. Once the polygons were created I put those into SQL Server (adding District and Sector fields for later use) so I could work on them further in QGIS.

The next issue was that the polygons created extended far beyond the shores of the UK. This required the use of the ‘Clip vector by mask layer’ tool in QGIS, using a UK border polygon layer as the mask (note to self for next time I do this – this takes hours to complete). Then the postcode units needed to be merged into districts and sectors. The ‘Dissolve’ tool in QGIS can be used for this, using the District and Sector fields as the dissolve field.

I then had two KML files, one with all postcode districts and one with all postcode sectors. I then wrote some C# code to split the files apart so the individual districts and sectors can be displayed and downloaded.

Land Registry data for February 2019

I’ve just uploaded the latest house price data to my website. Not much to report (2% annual inflation still), but I do have some prettier charts

Land Registry data for January 2019

I’ve uploaded the latest property sales data from the Land Registry to my website. Not much to report, house price inflation remains at 2%, transactions remain flat

UK postcode data for February 2019

I have uploaded the latest UK postcode data from the ONS to my website. I’ve run my usual sanity checks and everything looks OK but let me know if anything appears amiss. Enjoy all the 2,611,698 postcodes (alive and dead)

UK station usage 2018

I’ve added 2018 station usage date for UK stations to my website. Enjoy

Land Registry data for Dec 2018

I’ve uploaded the latest house price data from the Land Registry to my website. Currently every month feels like Groundhog Day, prices continue to rise at about 2%, the number of transactions is stable.

Land Registry house price data November 2018

The good people at the Land Registry were probably taking a well earned vacation, so the data arrived a little later than normal but house price data for England and Wales for November 2018 is now on my site. Not much to report, annual inflation is still sitting at about 2%, the number of transactions are holding steady. Maybe things will get interesting in March, I hear something might be happening around then.

Free for all

As mentioned in a previous post, Google have started charging me for some more Google Maps related things but have also given me some free credit to tide me over. In fact, they think my future usage of Google Maps is going to be so high, they have given me an absolutely huge amount of credits. Since I can’t actually do anything with those credits other than spend them on the Google Maps API, I have removed the requirement for a Google Maps API key on the pages that previously required one. This will remain in place until either my credits run out or we’re close to the end of January. So please go crazy. Happy Christmas.

Important note – if the API key field contains your API key, your API key will still be used!

Land Registry house price data for October 2018

The latest house price data from the Land Registry is now on my site. House prices continue to rise slowly, although trending lower currently.