Land Registry data for June 2022

I'm uploading the latest property data from the Land Registry to my website. Whilst the price of everything else is going through the roof, the price of houses have started going down for the first time in about ten years. The main reason for this is something I hadn't noticed before. Just before the Stamp Duty holiday ended, sales volumes and prices went ballistic. Those figures have now dropped out of the annual price calculations leading to a big drop in prices. The architect of this classic boom and bust could well be our next PM.

Land Registry data for May 2022

I've uploaded the latest property sales data to my website. House price inflation seems to be bouncing around close to zero, which I guess is why the government is talking about 50 year mortgages... 

Land Registry property data for April 2022

I've uploaded the latest property data for England and Wales to my website. The annual figures still show sales falling with prices flat and likely to turn negative next month.

UK postcode data update

I've uploaded the latest UK postcode data to my website. Some people have been asking about the ward boundary changes. These are included in the update but are not complete so I would approach with caution. I made some manual changes so that all postcodes have a ward associated but these will be the old wards in some cases.

From the ONSPD user guide

This release of the NHSPD includes the 2022 ward boundary changes. However, some change orders have yet to be included in OS Boundary-Line including The Cumbria (Structural Changes) Order 2022, The North Yorkshire (Structural Changes) Order 2022 and The Somerset (Structural Changes) Order 2022. Therefore, ward codes E05014171–E05014393 are not yet included. Also, Welsh wards W05001616 –W05001626 in Merthyr Tydfil are yet to be included and Sirhowy ward in Blaenau Gwent should be W05001268 rather than W05000764. In Scotland there is a duplicated ward code – S13003134. This should be S13003134 – Stepps, Chryston and Muirhead in North Lanarkshire and S13002608 – An Taobh Siar agus Nis in Na h-Eileanan Siar

House price for England and Wales March 2022

I'm currently uploading the latest house price data to my website. I really need to improve the performance of the import, it now takes days to complete, but it should be complete some time soon. 

There's a definite trend appearing in the data since last summer, sales are falling and prices have peaked. Annual house price inflation is still positive but it looks like it will go negative very shortly, unless the government intervene again. It's also noticeable that the price drops are mostly being driven by the price of flats, although I have no insight into why that would be the case.

Converting .ashx handlers to .NET 5/6

I've been trying to figure out how to convert my site to .NET 5 or 6 for a long while. The pages are written in PHP (because that was the only thing available to me on my web host at the time) but there are many .ashx handlers that were added when I originally decided to move to ASP.NET but never got round to completing the job. But the .NET Framework isn't going to be improved or added to in the future so moving to .NET 5/6 needs to happen at some point. It's not a simple process, the differences are fairly major, but I think I've finally figured out a path to making the move.

My main concern was that I wanted to have a common code base so I can reuse code and continue to work on my current site whilst I set up the updated one, rather than rewriting the whole thing from scratch. A rewrite would certainly be simpler but the switch over would almost certainly be a disaster.

The first step was to work on assemblies used by the site. The Portability Analyzer helped here in figuring out how portable my code was (although it's not generally too helpful in explaining what to do with APIs that are not available in .NET 5). Fortunately it wasn't too difficult to move .NET Coords to .NET Standard 2, meaning it could be used from old and new .NET. I used the .NET Upgrade Assistant to convert to the latest SDK project format. Luckily there were cross platform versions of the third party assemblies I use.

One other thing I did was rip out Entity Framework, which I've never been able to get along with and move from the Oracle MySql ADO.NET connector to MySqlConnector. Like everything from Oracle on Windows, their MySql connector is a complete dumpster fire.

Once that was done, I decided the next step was to move all my .ashx handler code into a new assembly. So every .ashx file now looks something like

<%@ WebHandler Language="C#" Class="DoogalCode.AdministrativeAreasCSV" %>

I configured that assembly to target .NET 4.8 and .NET 5, since there would have to be differences between them. At this point, the .NET 4.8 assembly compiled fine (although with many warnings about using things from System.Web even though that's perfectly fine in .NET 4.8!) and the .NET 5 assembly failed with a gazillion errors. 

The next step was to build the .NET 5 assembly without errors. I decided the simplest way to do that was to build .NET 5 only versions of all the things the compiler was complaining about. This meant things like HttpContext, HttpRequest, HttpResponse and IHttpHandler. The properties and methods didn't do anything except throw an exception, I didn't want the code to run, just compile. I'd figure out the details of  how to implement those methods as I went through each handler and got them working in .NET 5.

Once I'd done that, I needed a .NET web site. Since my front end is written in PHP, I decided to take a look at PeachPie, which claims to be a PHP compiler for .NET. And I am mightily impressed. I created a new PeachPie project, copied my PHP files across and it just worked. I had my website running in .NET 5 in a few minutes. Admittedly I don't do anything very complicated in my PHP, so I can't be sure it's perfect but it's certainly perfect for my needs.

But I still had a bunch of .ashx handlers to get working in .NET 5. I added a method to my website startup class that let me use my dummy IHttpHandler and HttpContext classes to call into my handlers

private static void MapHandler<THandler>(IApplicationBuilder app, string path) where THandler : IHttpHandler, new()

{

  app.Map(path, (app2) =>

  {

    app2.Run(async context =>

    {

      var handler = new THandler();

      await Task.Run(() =>

      {

        handler.ProcessRequest(new DoogalCode.HttpContext(context));

      });

    });

  });

}

Then for each handler I mapped URL paths to the handler via this method, like so

MapHandler<CountiesCSV>(app, "/CountiesCSV");

Now I could call my handlers and start to fix up the methods in my dummy classes. Which is where I am currently, going through each handler fixing issues as I find them, whilst still being able to work on my live website. Once that is done and has gone live, I may try to figure out how to finally move my PHP code to proper .NET

Addendum - This all worked out really well. I was able to retarget all my classic .NET code to .NET 5, whilst continuing to keep the site running. Switching over to the new website was relatively painless, with little downtime. Moving to .NET 6 was straightforward. I was then able to finally start replacing PHP pages with .NET Razor pages, which I'm currently working through. I've also recently learned that Microsoft are developing their own adapters for System.Web.HttpContext etc to make the transition easier, which I wish I'd known sooner!

House price data for England and Wales February 2022

I've uploaded the latest Land Registry data to my website. I've also added a "Sale Type" column to the data which shows if the sale is a non-standard sale. These include transfers under a power of sale/repossessions, buy-to-lets, transfers to non-private individuals and sales where the property type is classed as ‘Other’. The aggregated data now ignores non-standard sales, although it has not had a major effect.

Land Registry data for Jan 2022

I've uploaded the latest Land Registry data for England and Wales to my website. No analysis, there's more important things to think about.

Distance to the sea for all postcodes

I've added the approximate distance to the sea for every postcode. This is available on the individual postcode pages and as part of the CSV downloads

Distance to a line

Probably only of use to me, but I've added a page that lets you find the shortest distance between a postcode and a straight line

Distance to a line (doogal.co.uk)

UK postcode data update

I've uploaded the latest ONS postcode data to my website. The data has passed my sanity tests, but let me know if you spot anything amiss. The CSV downloads now include a UPRNs column, which is a comma separated list of the UPRNs in the postcode

Land Registry data for December 2021

I've uploaded the latest house price data for England and Wales to my website. And it's as you were, with lower sales and slowing inflation in prices. 

.NET Coords in .NET Core

.NET Coords is now a .NET Standard 2.0 project, meaning it can be used in .NET Framework 4.8 and .NET Core projects

UPRN data

I've uploaded UPRN (Unique Property Reference Number) data for the UK to my website. They are only currently available as a link from viewing a particular postcode (an example) but do let me know if it would be useful to provide them in some other way.

Unfortunately the freely available data doesn't include the thing that would be make this data exceedingly useful, the address. I doubt that is likely to happen since there are many companies whose business is based on selling that data

One other major problem is the open data includes old UPRNs but no indication that the UPRN is no longer active, hopefully this will be addressed in a future release.

There's more information on this data here which covers other issues with it.

C# code to generate a segment FIT file

Someone asked about getting hold of the code to generate FIT files from Strava segments. It's not possible to provide standalone code since it's has dependencies on the rest of my website and a Strava library, but here's what I use which may give people an idea of how to use the rather confusing Garmin FIT API

using System;
using System.Collections.Generic;
using System.IO;
using System.Web;
using Dynastream.Fit;
using Newtonsoft.Json.Linq;
using Strava.Common;
using Strava.Segments;
using Strava.Streams;

public static class FitGenerator
{ 
  private static void AddLeaderEntry(SegmentPointMesg newRecord, int goal, float currentDistance, 
    float totalDistance, byte goalIndex)  
  {
    if (goal != 0)
    {
      var currentGoalTime = (float)Math.Floor(currentDistance / totalDistance * goal);
      newRecord.SetLeaderTime(goalIndex, currentGoalTime);
    }
  }

  private static SegmentLeaderboardEntryMesg GetLeaderboard(int goal, byte goalIndex, 
    SegmentLeaderboardType type, string name)
  {
    var goalLeaderboard = new SegmentLeaderboardEntryMesg();
    if (goal != 0)
    {
      goalLeaderboard.SetMessageIndex(goalIndex);
      goalLeaderboard.SetSegmentTime(goal);
      goalLeaderboard.SetType(type);
      if (!string.IsNullOrEmpty(name))
        goalLeaderboard.SetName(name);
    }

    return goalLeaderboard;
  }

  private static int LatOrLngToSemicircle(double latOrLng)
  {
    return (int)(latOrLng * (Math.Pow(2, 31) / 180));
  }

  private static bool HasPr(List<SegmentStream> pr, Segment seg)
  {
    if (pr != null)
      return true;
    if (seg.AthleteSegmentStats != null && seg.AthleteSegmentStats.PrElapsedTime != null)
      return true;

    return false;
  }

  public static byte[] Generate(Segment seg, int goal, string goalName, int rival, string rivalName, 
    int challenger, string challengerName)
  {
    List<SegmentStream> altitude = null;
    try
    {
      StravaBaseHandler.TryStravaRequest((client) =>
      {
        altitude = client.Streams.GetSegmentStream(seg.Id.ToString(), 
          SegmentStreamType.Altitude | SegmentStreamType.LatLng);
      });
    }
    catch (Exception ex)
    {
      throw new Exception("Failed to get stream for segment " + seg.Id, ex);
    }

    // get all the streams we are interested in
    SegmentStream locationStream = null;
    SegmentStream altitudeStream = null;
    SegmentStream distanceStream = null;

    locationStream = SegmentStream.GetStream(altitude, StreamType.LatLng);
    altitudeStream = SegmentStream.GetStream(altitude, StreamType.Altitude);
    distanceStream = SegmentStream.GetStream(altitude, StreamType.Distance);

    // grab user's best time
    List<SegmentStream> pr = null;
    var athlete = HttpContext.Current.Request.Cookies["athlete"];
    if (athlete != null)
    {
      try
      {
        pr = StravaBaseHandler.GetSegmentPr(seg.Id.ToString(), athlete.Value);
      }
      catch
      {
        // do nothing
      }
    }

    // Generate some FIT messages
    // Every FIT file MUST contain a 'File ID' message as the first message
    var fileIdMesg = new FileIdMesg(); 
    var records = new List<SegmentPointMesg>();

    fileIdMesg.SetType(Dynastream.Fit.File.Segment);
    fileIdMesg.SetManufacturer(Manufacturer.Strava);  // Types defined in the profile are available
    fileIdMesg.SetProduct(65534);
    fileIdMesg.SetTimeCreated(new Dynastream.Fit.DateTime(System.DateTime.Now));
    fileIdMesg.SetSerialNumber(1);
    fileIdMesg.SetNumber(1);

    var fileCreator = new FileCreatorMesg();
    fileCreator.SetHardwareVersion(0);
    fileCreator.SetSoftwareVersion(0);

    // segment ID
    var segmentId = new SegmentIdMesg();
    segmentId.SetName(seg.Name);
    segmentId.SetEnabled(Bool.True);

    if (seg.ActivityType == "Run")
      segmentId.SetSport(Sport.Running);
    else
      segmentId.SetSport(Sport.Cycling);

    byte prIndex = 0;
    byte goalIndex = 1;
    byte rivalIndex = 2;
    byte challengerIndex = 3;
    byte komIndex = 4;
    byte qomIndex = 5;

    if (!HasPr(pr, seg))
    {
      goalIndex--;
      rivalIndex--;
      challengerIndex--;
      komIndex--;
      qomIndex--;
    }

    if (goal == 0)
    {
      rivalIndex--;
      challengerIndex--;
      komIndex--;
      qomIndex--;
    }

    if (rival == 0)
    {
      challengerIndex--;
      komIndex--;
      qomIndex--;
    }

    if (challenger == 0)
    {
      komIndex--;
      qomIndex--;
    }

    if (seg.KOM() == 0)
    {
      qomIndex--;
    }

    segmentId.SetSelectionType(SegmentSelectionType.Starred);
    segmentId.SetUuid(Guid.NewGuid().ToByteArray());
    if (HasPr(pr, seg))
      segmentId.SetDefaultRaceLeader(prIndex);

    if (seg.Map.Polyline == "")
      throw new Exception("The segment has no map data (which is somewhat unexpected), " + 
        "I can't generate the FIT file");

    var points = PolylineDecoder.Decode(seg.Map.Polyline);

    // figure out SW and NE
    double swLat = 1000;
    double swLong = 1000;
    double neLat = -1000;
    double neLong = -1000;
    foreach (var pt in points)
    {
      swLat = Math.Min(swLat, pt.Latitude);
      swLong = Math.Min(swLong, pt.Longitude);
      neLat = Math.Max(neLat, pt.Latitude);
      neLong = Math.Max(neLong, pt.Longitude);
    }

    // segment info
    var lap = new SegmentLapMesg();
    lap.SetUuid(Guid.NewGuid().ToByteArray());
    lap.SetTotalDistance(seg.Distance);
    lap.SetTotalAscent((ushort)seg.TotalElevationGain);
    lap.SetSwcLat(LatOrLngToSemicircle(swLat));
    lap.SetSwcLong(LatOrLngToSemicircle(swLong));
    lap.SetNecLat(LatOrLngToSemicircle(neLat));
    lap.SetNecLong(LatOrLngToSemicircle(neLong));
    lap.SetMessageIndex(1);
    lap.SetStartPositionLat(LatOrLngToSemicircle(points[0].Latitude));
    lap.SetStartPositionLong(LatOrLngToSemicircle(points[0].Longitude));
    lap.SetEndPositionLat(LatOrLngToSemicircle(points[points.Count - 1].Latitude));
    lap.SetEndPositionLong(LatOrLngToSemicircle(points[points.Count - 1].Longitude));

    // goal entry
    var goalLeaderboard = GetLeaderboard(goal, goalIndex, SegmentLeaderboardType.Goal, goalName);
    var rivalLeaderboard = GetLeaderboard(rival, rivalIndex, SegmentLeaderboardType.Rival, rivalName);
    var challengerLeaderboard = 
      GetLeaderboard(challenger, challengerIndex, SegmentLeaderboardType.Challenger, challengerName);
    var komLeaderboard = GetLeaderboard(seg.KOM(), komIndex, SegmentLeaderboardType.Kom, "");
    var qomLeaderboard = GetLeaderboard(seg.QOM(), qomIndex, SegmentLeaderboardType.Qom, "");

    SegmentStream prDistanceStream = null;
    SegmentStream prTimeStream = null;
    if (pr != null)
    {
      prDistanceStream = SegmentStream.GetStream(pr, StreamType.Distance);
      prTimeStream = SegmentStream.GetStream(pr, StreamType.Time);
    }

    SegmentLeaderboardEntryMesg prLeaderboard = null;
    if (HasPr(pr, seg))
    {
      prLeaderboard = new SegmentLeaderboardEntryMesg();
      prLeaderboard.SetMessageIndex(prIndex);
      var prTime = seg.AthleteSegmentStats.PrElapsedTime;
      prLeaderboard.SetSegmentTime(prTime);
      prLeaderboard.SetType(SegmentLeaderboardType.Pr);
    }

    for (var i = 0; i < locationStream.Data.Count; i++)
    {
      var newRecord = new SegmentPointMesg();
      var latLng = locationStream.Data[i];
      if (latLng != null)
      {
        var intLat = LatOrLngToSemicircle(Convert.ToDouble(((JArray)(latLng)).First));
        newRecord.SetPositionLat(intLat);
        var intLng = LatOrLngToSemicircle(Convert.ToDouble(((JArray)(latLng)).Last));
        newRecord.SetPositionLong(intLng);
      }

      if (altitudeStream != null)
        newRecord.SetAltitude(Convert.ToSingle(altitudeStream.Data[i]));
      var currentDistance = Convert.ToSingle(distanceStream.Data[i]);
      newRecord.SetDistance(currentDistance);
      newRecord.SetMessageIndex((ushort)i);

      if (HasPr(pr, seg))
      {
        if (pr != null)
        {
          // add PR time
          // find index of the PR item that is at or beyond current distance
          var pos = prDistanceStream.Data.Count - 1;
          for (var j = 0; j < prDistanceStream.Data.Count; j++)
          {
            if (Convert.ToSingle(prDistanceStream.Data[j]) - 
              Convert.ToSingle(prDistanceStream.Data[0]) >= currentDistance)
            {
              pos = j;
              break;
            }
          }
          // find time at that distance
          if (pos > 0)
          {
            var prStartDistance = Convert.ToSingle(prDistanceStream.Data[pos - 1]) - 
              Convert.ToSingle(prDistanceStream.Data[0]);
            var prEndDistance = Convert.ToSingle(prDistanceStream.Data[pos]) - 
              Convert.ToSingle(prDistanceStream.Data[0]);
            var fraction = (currentDistance - prStartDistance) / (prEndDistance - prStartDistance);

            var prTime = Convert.ToSingle(prTimeStream.Data[pos - 1]) + fraction * 
              (Convert.ToSingle(prTimeStream.Data[pos]) - Convert.ToSingle(prTimeStream.Data[pos - 1]));

            newRecord.SetLeaderTime(prIndex, 
              (float)Math.Floor(prTime - Convert.ToSingle(prTimeStream.Data[0])));
          }
          else
          {
            newRecord.SetLeaderTime(prIndex, 0);
          }
        }
        else
        {
          AddLeaderEntry(newRecord, seg.AthleteSegmentStats.PrElapsedTime.Value, currentDistance, 
            seg.Distance, prIndex);
        }
      }

      // add goal time
      AddLeaderEntry(newRecord, goal, currentDistance, seg.Distance, goalIndex);
      AddLeaderEntry(newRecord, rival, currentDistance, seg.Distance, rivalIndex);
      AddLeaderEntry(newRecord, challenger, currentDistance, seg.Distance, challengerIndex);
      AddLeaderEntry(newRecord, seg.KOM(), currentDistance, seg.Distance, komIndex);
      AddLeaderEntry(newRecord, seg.QOM(), currentDistance, seg.Distance, qomIndex);

      records.Add(newRecord);
    }

    // Create file encode object
    var encodeDemo = new Encode(ProtocolVersion.V20);

    using (var fitDest = new MemoryStream())
    {

      // Write our header
      encodeDemo.Open(fitDest);

      // Encode each message, a definition message is automatically generated and output if necessary
      encodeDemo.Write(fileIdMesg);
      encodeDemo.Write(fileCreator);
      encodeDemo.Write(segmentId);
      if (prLeaderboard != null)
        encodeDemo.Write(prLeaderboard);
      if (goal != 0)
        encodeDemo.Write(goalLeaderboard);
      if (rival != 0)
        encodeDemo.Write(rivalLeaderboard);
      if (challenger != 0)
        encodeDemo.Write(challengerLeaderboard);
      if (seg.KOM() != 0)
        encodeDemo.Write(komLeaderboard);
      if (seg.QOM() != 0)
        encodeDemo.Write(qomLeaderboard);

      encodeDemo.Write(lap);
      encodeDemo.Write(records);
      encodeDemo.Close();

      fitDest.Flush();
      fitDest.Position = 0;
      return fitDest.ToArray();
    }
  }
}

Land Registry November 2021

My server is churning through the latest Land Registry data (for November 2021) and the top level figures are now available on my website. Prices continue upwards although at a slower pace, the number of sales are still subdued after the end of the Stamp Duty holiday

Update to .NET Coordinates

I've released a new version of .NET Coordinates to add support for the Irish Transverse Mercator coordinate system. This was introduced 20 years ago to replace the Irish National Grid and I only heard of it about a week ago... I've also made some changes under the hood to make adding new coordinate systems easier in future so if your favourite isn't available, let me know.

Distance to sea improvements

I've been showing some love to the Distance to Sea page. I've improved the precision of the coastline I'm using for the calculation, although it's still not super accurate since it would be really slow to load. I've also improved the algorithm used to calculate the distance so it should give more sensible results. I've also made a few other minor tweaks that should make it easier to use.

Land Registry data for October 2021

I've uploaded the latest Land Registry data to my website. It's been a few months since the end of the Stamp Duty holiday and it's now clear the number of sales has fallen quite dramatically since then. Prices are still increasing but at a slower rate than previously.

UK postcode data update

I've uploaded the November 2021 ONS postcode data to the site. It all looks good but let me know of any problems