UPRN data

I've uploaded UPRN (Unique Property Reference Number) data for the UK to my website. They are only currently available as a link from viewing a particular postcode (an example) but do let me know if it would be useful to provide them in some other way.

Unfortunately the freely available data doesn't include the thing that would be make this data exceedingly useful, the address. I doubt that is likely to happen since there are many companies whose business is based on selling that data

One other major problem is the open data includes old UPRNs but no indication that the UPRN is no longer active, hopefully this will be addressed in a future release.

There's more information on this data here which covers other issues with it.

C# code to generate a segment FIT file

Someone asked about getting hold of the code to generate FIT files from Strava segments. It's not possible to provide standalone code since it's has dependencies on the rest of my website and a Strava library, but here's what I use which may give people an idea of how to use the rather confusing Garmin FIT API

using System;
using System.Collections.Generic;
using System.IO;
using System.Web;
using Dynastream.Fit;
using Newtonsoft.Json.Linq;
using Strava.Common;
using Strava.Segments;
using Strava.Streams;

public static class FitGenerator
{ 
  private static void AddLeaderEntry(SegmentPointMesg newRecord, int goal, float currentDistance, 
    float totalDistance, byte goalIndex)  
  {
    if (goal != 0)
    {
      var currentGoalTime = (float)Math.Floor(currentDistance / totalDistance * goal);
      newRecord.SetLeaderTime(goalIndex, currentGoalTime);
    }
  }

  private static SegmentLeaderboardEntryMesg GetLeaderboard(int goal, byte goalIndex, 
    SegmentLeaderboardType type, string name)
  {
    var goalLeaderboard = new SegmentLeaderboardEntryMesg();
    if (goal != 0)
    {
      goalLeaderboard.SetMessageIndex(goalIndex);
      goalLeaderboard.SetSegmentTime(goal);
      goalLeaderboard.SetType(type);
      if (!string.IsNullOrEmpty(name))
        goalLeaderboard.SetName(name);
    }

    return goalLeaderboard;
  }

  private static int LatOrLngToSemicircle(double latOrLng)
  {
    return (int)(latOrLng * (Math.Pow(2, 31) / 180));
  }

  private static bool HasPr(List<SegmentStream> pr, Segment seg)
  {
    if (pr != null)
      return true;
    if (seg.AthleteSegmentStats != null && seg.AthleteSegmentStats.PrElapsedTime != null)
      return true;

    return false;
  }

  public static byte[] Generate(Segment seg, int goal, string goalName, int rival, string rivalName, 
    int challenger, string challengerName)
  {
    List<SegmentStream> altitude = null;
    try
    {
      StravaBaseHandler.TryStravaRequest((client) =>
      {
        altitude = client.Streams.GetSegmentStream(seg.Id.ToString(), 
          SegmentStreamType.Altitude | SegmentStreamType.LatLng);
      });
    }
    catch (Exception ex)
    {
      throw new Exception("Failed to get stream for segment " + seg.Id, ex);
    }

    // get all the streams we are interested in
    SegmentStream locationStream = null;
    SegmentStream altitudeStream = null;
    SegmentStream distanceStream = null;

    locationStream = SegmentStream.GetStream(altitude, StreamType.LatLng);
    altitudeStream = SegmentStream.GetStream(altitude, StreamType.Altitude);
    distanceStream = SegmentStream.GetStream(altitude, StreamType.Distance);

    // grab user's best time
    List<SegmentStream> pr = null;
    var athlete = HttpContext.Current.Request.Cookies["athlete"];
    if (athlete != null)
    {
      try
      {
        pr = StravaBaseHandler.GetSegmentPr(seg.Id.ToString(), athlete.Value);
      }
      catch
      {
        // do nothing
      }
    }

    // Generate some FIT messages
    // Every FIT file MUST contain a 'File ID' message as the first message
    var fileIdMesg = new FileIdMesg(); 
    var records = new List<SegmentPointMesg>();

    fileIdMesg.SetType(Dynastream.Fit.File.Segment);
    fileIdMesg.SetManufacturer(Manufacturer.Strava);  // Types defined in the profile are available
    fileIdMesg.SetProduct(65534);
    fileIdMesg.SetTimeCreated(new Dynastream.Fit.DateTime(System.DateTime.Now));
    fileIdMesg.SetSerialNumber(1);
    fileIdMesg.SetNumber(1);

    var fileCreator = new FileCreatorMesg();
    fileCreator.SetHardwareVersion(0);
    fileCreator.SetSoftwareVersion(0);

    // segment ID
    var segmentId = new SegmentIdMesg();
    segmentId.SetName(seg.Name);
    segmentId.SetEnabled(Bool.True);

    if (seg.ActivityType == "Run")
      segmentId.SetSport(Sport.Running);
    else
      segmentId.SetSport(Sport.Cycling);

    byte prIndex = 0;
    byte goalIndex = 1;
    byte rivalIndex = 2;
    byte challengerIndex = 3;
    byte komIndex = 4;
    byte qomIndex = 5;

    if (!HasPr(pr, seg))
    {
      goalIndex--;
      rivalIndex--;
      challengerIndex--;
      komIndex--;
      qomIndex--;
    }

    if (goal == 0)
    {
      rivalIndex--;
      challengerIndex--;
      komIndex--;
      qomIndex--;
    }

    if (rival == 0)
    {
      challengerIndex--;
      komIndex--;
      qomIndex--;
    }

    if (challenger == 0)
    {
      komIndex--;
      qomIndex--;
    }

    if (seg.KOM() == 0)
    {
      qomIndex--;
    }

    segmentId.SetSelectionType(SegmentSelectionType.Starred);
    segmentId.SetUuid(Guid.NewGuid().ToByteArray());
    if (HasPr(pr, seg))
      segmentId.SetDefaultRaceLeader(prIndex);

    if (seg.Map.Polyline == "")
      throw new Exception("The segment has no map data (which is somewhat unexpected), " + 
        "I can't generate the FIT file");

    var points = PolylineDecoder.Decode(seg.Map.Polyline);

    // figure out SW and NE
    double swLat = 1000;
    double swLong = 1000;
    double neLat = -1000;
    double neLong = -1000;
    foreach (var pt in points)
    {
      swLat = Math.Min(swLat, pt.Latitude);
      swLong = Math.Min(swLong, pt.Longitude);
      neLat = Math.Max(neLat, pt.Latitude);
      neLong = Math.Max(neLong, pt.Longitude);
    }

    // segment info
    var lap = new SegmentLapMesg();
    lap.SetUuid(Guid.NewGuid().ToByteArray());
    lap.SetTotalDistance(seg.Distance);
    lap.SetTotalAscent((ushort)seg.TotalElevationGain);
    lap.SetSwcLat(LatOrLngToSemicircle(swLat));
    lap.SetSwcLong(LatOrLngToSemicircle(swLong));
    lap.SetNecLat(LatOrLngToSemicircle(neLat));
    lap.SetNecLong(LatOrLngToSemicircle(neLong));
    lap.SetMessageIndex(1);
    lap.SetStartPositionLat(LatOrLngToSemicircle(points[0].Latitude));
    lap.SetStartPositionLong(LatOrLngToSemicircle(points[0].Longitude));
    lap.SetEndPositionLat(LatOrLngToSemicircle(points[points.Count - 1].Latitude));
    lap.SetEndPositionLong(LatOrLngToSemicircle(points[points.Count - 1].Longitude));

    // goal entry
    var goalLeaderboard = GetLeaderboard(goal, goalIndex, SegmentLeaderboardType.Goal, goalName);
    var rivalLeaderboard = GetLeaderboard(rival, rivalIndex, SegmentLeaderboardType.Rival, rivalName);
    var challengerLeaderboard = 
      GetLeaderboard(challenger, challengerIndex, SegmentLeaderboardType.Challenger, challengerName);
    var komLeaderboard = GetLeaderboard(seg.KOM(), komIndex, SegmentLeaderboardType.Kom, "");
    var qomLeaderboard = GetLeaderboard(seg.QOM(), qomIndex, SegmentLeaderboardType.Qom, "");

    SegmentStream prDistanceStream = null;
    SegmentStream prTimeStream = null;
    if (pr != null)
    {
      prDistanceStream = SegmentStream.GetStream(pr, StreamType.Distance);
      prTimeStream = SegmentStream.GetStream(pr, StreamType.Time);
    }

    SegmentLeaderboardEntryMesg prLeaderboard = null;
    if (HasPr(pr, seg))
    {
      prLeaderboard = new SegmentLeaderboardEntryMesg();
      prLeaderboard.SetMessageIndex(prIndex);
      var prTime = seg.AthleteSegmentStats.PrElapsedTime;
      prLeaderboard.SetSegmentTime(prTime);
      prLeaderboard.SetType(SegmentLeaderboardType.Pr);
    }

    for (var i = 0; i < locationStream.Data.Count; i++)
    {
      var newRecord = new SegmentPointMesg();
      var latLng = locationStream.Data[i];
      if (latLng != null)
      {
        var intLat = LatOrLngToSemicircle(Convert.ToDouble(((JArray)(latLng)).First));
        newRecord.SetPositionLat(intLat);
        var intLng = LatOrLngToSemicircle(Convert.ToDouble(((JArray)(latLng)).Last));
        newRecord.SetPositionLong(intLng);
      }

      if (altitudeStream != null)
        newRecord.SetAltitude(Convert.ToSingle(altitudeStream.Data[i]));
      var currentDistance = Convert.ToSingle(distanceStream.Data[i]);
      newRecord.SetDistance(currentDistance);
      newRecord.SetMessageIndex((ushort)i);

      if (HasPr(pr, seg))
      {
        if (pr != null)
        {
          // add PR time
          // find index of the PR item that is at or beyond current distance
          var pos = prDistanceStream.Data.Count - 1;
          for (var j = 0; j < prDistanceStream.Data.Count; j++)
          {
            if (Convert.ToSingle(prDistanceStream.Data[j]) - 
              Convert.ToSingle(prDistanceStream.Data[0]) >= currentDistance)
            {
              pos = j;
              break;
            }
          }
          // find time at that distance
          if (pos > 0)
          {
            var prStartDistance = Convert.ToSingle(prDistanceStream.Data[pos - 1]) - 
              Convert.ToSingle(prDistanceStream.Data[0]);
            var prEndDistance = Convert.ToSingle(prDistanceStream.Data[pos]) - 
              Convert.ToSingle(prDistanceStream.Data[0]);
            var fraction = (currentDistance - prStartDistance) / (prEndDistance - prStartDistance);

            var prTime = Convert.ToSingle(prTimeStream.Data[pos - 1]) + fraction * 
              (Convert.ToSingle(prTimeStream.Data[pos]) - Convert.ToSingle(prTimeStream.Data[pos - 1]));

            newRecord.SetLeaderTime(prIndex, 
              (float)Math.Floor(prTime - Convert.ToSingle(prTimeStream.Data[0])));
          }
          else
          {
            newRecord.SetLeaderTime(prIndex, 0);
          }
        }
        else
        {
          AddLeaderEntry(newRecord, seg.AthleteSegmentStats.PrElapsedTime.Value, currentDistance, 
            seg.Distance, prIndex);
        }
      }

      // add goal time
      AddLeaderEntry(newRecord, goal, currentDistance, seg.Distance, goalIndex);
      AddLeaderEntry(newRecord, rival, currentDistance, seg.Distance, rivalIndex);
      AddLeaderEntry(newRecord, challenger, currentDistance, seg.Distance, challengerIndex);
      AddLeaderEntry(newRecord, seg.KOM(), currentDistance, seg.Distance, komIndex);
      AddLeaderEntry(newRecord, seg.QOM(), currentDistance, seg.Distance, qomIndex);

      records.Add(newRecord);
    }

    // Create file encode object
    var encodeDemo = new Encode(ProtocolVersion.V20);

    using (var fitDest = new MemoryStream())
    {

      // Write our header
      encodeDemo.Open(fitDest);

      // Encode each message, a definition message is automatically generated and output if necessary
      encodeDemo.Write(fileIdMesg);
      encodeDemo.Write(fileCreator);
      encodeDemo.Write(segmentId);
      if (prLeaderboard != null)
        encodeDemo.Write(prLeaderboard);
      if (goal != 0)
        encodeDemo.Write(goalLeaderboard);
      if (rival != 0)
        encodeDemo.Write(rivalLeaderboard);
      if (challenger != 0)
        encodeDemo.Write(challengerLeaderboard);
      if (seg.KOM() != 0)
        encodeDemo.Write(komLeaderboard);
      if (seg.QOM() != 0)
        encodeDemo.Write(qomLeaderboard);

      encodeDemo.Write(lap);
      encodeDemo.Write(records);
      encodeDemo.Close();

      fitDest.Flush();
      fitDest.Position = 0;
      return fitDest.ToArray();
    }
  }
}

Land Registry November 2021

My server is churning through the latest Land Registry data (for November 2021) and the top level figures are now available on my website. Prices continue upwards although at a slower pace, the number of sales are still subdued after the end of the Stamp Duty holiday

Update to .NET Coordinates

I've released a new version of .NET Coordinates to add support for the Irish Transverse Mercator coordinate system. This was introduced 20 years ago to replace the Irish National Grid and I only heard of it about a week ago... I've also made some changes under the hood to make adding new coordinate systems easier in future so if your favourite isn't available, let me know.

Distance to sea improvements

I've been showing some love to the Distance to Sea page. I've improved the precision of the coastline I'm using for the calculation, although it's still not super accurate since it would be really slow to load. I've also improved the algorithm used to calculate the distance so it should give more sensible results. I've also made a few other minor tweaks that should make it easier to use.

Land Registry data for October 2021

I've uploaded the latest Land Registry data to my website. It's been a few months since the end of the Stamp Duty holiday and it's now clear the number of sales has fallen quite dramatically since then. Prices are still increasing but at a slower rate than previously.

UK postcode data update

I've uploaded the November 2021 ONS postcode data to the site. It all looks good but let me know of any problems

Constituency MPs

I've updated my constituency data to include the MP and their party. I've also improved the KML to use markers representing each party

Bus stops

Everybody loves buses, right? So everybody must also love bus stops. A while ago someone asked about adding the nearest bus stop to the postcode data alongside the nearest train station and I've since imported the NapTan bus stop data into my database. Nearest bus stops can now be seen when viewing individual postcodes, train stations and football stadiums.

What I haven't done is add the nearest bus stop to the postcodes data. I'm wondering if that's actually a useful thing to include since a bus stop a little further away may have more regular services or the buses may go in a more useful direction. But if anyone does want this data or wants other bus stop related data, let me know

Land Registry data for September 2021

I've uploaded the latest house price data from the Land Registry to my website. Prices are still rising although the rate of increase seems to have peaked

Welsh counties

I've added Welsh preserved counties to the counties data. If anybody knows of any KML files of the boundaries of these counties, I'd be interested in reusing the data

Find postcodes improvement

A minor update to the find postcodes page, you can now enter a latitude/longitude rather than clicking on the map

Find postcodes (doogal.co.uk) 

Land Registry data for Aug 2021

A bit later than usual this month due to packing my daughter off to university, but the Land Registry data for August is now being uploaded to my website. Covid and the end of the Stamp Duty holiday continue to make it difficult to draw any conclusions from the data

NUTS / ITL data now available

NUTS is yet another way of dividing countries into smaller subdivisions. It's been renamed to ITL since Brexit, although the defined areas currently remain the same.

I've added level 2 and level 3 data to the main postcode CSV downloads (NUTS level 1 data was already available as the Region column). If you'd like it available in any other formats, let me know.

UK Postcodes (doogal.co.uk)

Loading and saving shapes

I've made a little improvement to the shape generator page so you can load and save multiple collections of shapes. That means I can now use it for what I want to use it for!

Map shapes generator (doogal.co.uk)

Updated built up areas CSV download

The CSV download for built up areas now includes population, households and county data

Built up areas (doogal.co.uk)

Population and households data is still from 2011 unfortunately, 2021 census data has not been released yet

Updated map shapes generator

I've updated the polyline generator to include support for adding markers

Map shapes generator (doogal.co.uk) 

The cycle lane to hell

It started with me noticing I wasn't far from the top of the leaderboard for number of roads covered in Claygate on Wandrer. Then came the realisation that the only way to top that leaderboard was to ride along some of the A3. A bit of a search of Google Maps showed what appeared to be a cycle lane

So somebody at some point thought it was a sensible idea to ride on the A3 at the Esher junction. And to be fair, it didn't look too bad, other than the laughably narrow cycle lane on the slip road. Although the traffic is fairly fast moving at that point, there's a hard shoulder, which I'm guessing is where cyclists who want to live are meant to ride. But the hard shoulder disappears when the road merges with the Kingston bypass and you can't exit onto that road. You also can't exit at the next junction, by which point the A3 has turned into a nasty, busy, suburban thoroughfare. It's only after passing that junction and avoiding the traffic coming on via the slip road that you can actually exit onto a local road. 

I've searched up and down the road on Street View trying to find the officially sanctioned route for cyclists but I can find no further signs for cyclists. Did they just forget to finish this bit of infrastructure or was the planner actively trying to kill people?

I did figure out a couple of possible ways to escape the A3 before it gets too unpleasant. There's a tunnel under the A3 with a path into Claygate. That does involve negotiating a barbed wire fence but is otherwise OK. 

The two times I've been down that path the tunnel has been flooded so you may not have much luck if you need to get through it.

The other option would be to hop over onto the Kingston bypass via what looks like an unofficial junction drivers have created when they've realised they can't get there via a road. That involves negotiating the cars merging from the Kingston bypass and then probably stopping to get onto the grass, but probably OK if there's not much traffic

Find postcodes improvement

I've made a small modification to the Find Postcodes page so you can now upload a KML file and see the shapes within it rather than having to upload and draw your own shape around it. Let me know if there any problems with it

Latest Land Registry data

I'm in the process of uploading the latest Land Registry data to my site for July 2021. Although the sales data won't be complete yet, the effect of the end of the Stamp Duty holiday for higher value properties already appears to be visible in sale prices, with a noticable increase in prices in June and a big reduction in July

Latest UK postcode data

I've just uploaded the latest ONS postcode data to my website. I've run my usual sanity checks, but let me know if something looks wrong

Strava route viewer now using new maps

The Strava route viewer is now using the same maps as the segment explorer. Again hopefully I've not broken anything other than IE support

Strava segment explorer with new maps

I’ve updated my Strava segment explorer page to use different maps. These maps include more tracks and paths and also show contour lines to give you more of an idea of the gradient you’ll be facing. Other than that, it should work just the same as before. Of course I may have introduced a bug or two so let me know if you have any problems. Unfortunately I can no longer support Internet Explorer with these maps.

Finding roads in a polygon

I’ve made some improvements to the Find Places page. Support for roads is now better since it displays the actual path of the road, rather than just the centre point. So if you question is ‘what are the roads in this polygon?’, you can now find the answer.

Now also available on Twitter

I’ve decided to start posting updates on Twitter for people who prefer that for notifications

Follow me here

Land Registry data for June 2021

I’ve uploaded the latest Land Registry to my website. Prices continue on their upward trend with the number of sales looking to have almost returned to their pre-pandemic level. Meanwhile, economists suggest the Stamp Duty holiday was a bad idea, with any savings being wiped out by price increases.

Property sales May 2021

I’ve uploaded the latest Land Registry data to my website. Transactions continue to fall, prices continue to rise…

Land Registry sales data for April 2021

Another month, another drop of property data. And another month showing a big increase in property prices. You can find the data here. Whether intentional or not, the Stamp Duty holiday has caused a house price boom.

UK postcode data May 2021

I’ve uploaded the latest postcode data from the ONS to my website. I’ve run the usual sanity checks but as always let me know if you spot anything amiss

Land Registry data for March 2021

I’ve uploaded the latest house price data from the Land Registry to my website. It seems the Stamp Duty holiday, put in place to save the housing market during the pandemic, has actually caused something of a house price boom. Time will tell what happens at the end of the holiday.

House price data for February 2021

I’ve uploaded the latest house price data for England and Wales from the Land Registry to my website. We’re still in the middle of a pandemic but house prices continue to defy gravity, almost certainly due to the stamp duty holiday. That is due to end in June so we’ll see what happens after that

Land registry sales data for January 2021

I’ve uploaded the latest data from the Land Registry to my website. House price inflation seems to be increasing and the number of sales seems to be heading back to pre-pandemic levels.

Updated UK postcode data

I’ve updated the UK postcode data on my website to use the February 2021 ONS release. It’s passed my usual sanity checks but if you see anything amiss, let me know.

Land Registry property data for December 2020

I’ve uploaded the latest property data for England and Wales to my website. I’ve also made some changes to the charts so they hopefully give a clearer picture of trends in sales and prices. I think the most striking thing I see is that the number of sales over the last year have dropped to levels last seen in the days after the global financial crash. But unlike those days, this hasn’t had the same effect on prices which have continued to rise. I guess that’s down to the removal of stamp duty, so it will be interesting to see what happens when that ends.

Land Registry data for November 2020

I’ve uploaded the latest house sales data from the Land Registry to my website.

Website updates

I always forget to announce when new features have been added to the site, so here’s a few things that have been added over the last few months

• Find places – A new page to find places and other points of interest in the UK (driven by the OS Open Names dataset)
• Added the latest usage data for British stations
• Added football stadiums in Northern Ireland

If you find any issues, let me know

Land Registry property sales data October 2020

I’ve uploaded the October 2020 Land Registry property sales data for England and Wales to my website. I really can’t think of anything to say about it!

Postcode data for November 2020

I’ve uploaded the latest UK postcode data from the ONS to my website. I have checked that it looks OK but I often miss something so let me know if you spot anything amiss

England and Wales house price data update

The latest update from the Land Registry for September 2020 is now live on my site. Data still seems to be slow in coming through so the average sale price and number of sales is still volatile. I’m guessing the numbers from the summer are now complete which show prices flatlining and sales volumes much lower than normal

Land Registry data for August 2020

The latest property data for England and Wales is now on my website. The number of sales has certainly fallen over the last few months, even though I’ve been hearing of a property boom. And I’m not seeing much evidence of prices rising much either

Land Registry data for July 2020

The latest house price data from the Land Registry is now on my website. Sales are still way down but it’s too early to see what affect the Stamp Duty holiday has had, I’d imagine sales will be up over the coming months

UK postcode data for August 2020

I’ve uploaded the latest postcode data from the ONS to my website. I’ve run my usual sanity checks and it all looks OK, but do let me know if you spot any anomalies

House price data June 2020

I’ve uploaded the Land Registry data for June 2020 to my website. I reported a couple of months ago that there had been 7 sales in April. This has now been revised up to over 20,000 sales. Whilst not as terrible as the initial figures suggested, that’s still much lower activity than we’ve been seeing over the last few years. And although there’s been a marked drop in the prices paid, those prices haven’t affected the annual averages yet.

Property sales May 2020

I’ve uploaded the latest Land Registry property data to my site. There’s a continued lack of sales being completed, although last month’s figures have been revised upwards from 7 to 279 sales. That is still dramatically lower than the usual number of sales

If you’re wondering why the current average sale price has dropped significantly but the annual change in price has increased, this is due to the tiny number of sales happening, the sales dropping out of the other end are making a much bigger difference to the stats.

UK postcode data for May 2020

The ONS have published their latest postcode data which is now uploaded to my site. As ever, I’ve run some sanity checks (codes for administrative areas have changed yet again!) and everything looks to be correct. Let me know if you see anything amiss

(lack of) property sales April 2020

What happens to property sales in the middle of a pandemic? Perhaps unsurprisingly they fall off a cliff. According to the Land Registry, there were 7 sales in England and Wales in April, although that will probably go up a little as the data comes in for future months. With such a small number of sales, the price statistics are fairly meaningless

Land Registry data for March 2020

In the before days, people bought and sold houses. I’ve uploaded the data for England and Wales from March to my website. Since the data is never complete for the previous month, it’s difficult to see at this point the effect of the lockdown, but I assume it will become more obvious next month.

Land Registry data for February 2020

It may feel like a lifetime ago, but in the not too distant past people were able to buy and sell houses. I’ve uploaded the latest Land Registry data to my website which is fairly uninteresting, with prices still ticking up slowly. I imagine next month’s data will look very different.

Property data for January 2020

I’ve uploaded the latest Land Registry data for England and Wales to my website, covering sales up to January 2020. Prices continue to creep up, number of sales are hovering around the level they’ve been at since 2013 (when Help To Buy was introduced)

UK postcode data for February 2020

I’ve uploaded the latest postcode data from the ONS to my website. I’ve run the usual sanity checks but do let me know if you spot anything amiss

Moving from a Garmin 810 to a Garmin 830

About 5 years ago, I decided I needed to buy myself a dedicated bike computer, rather than the phone I’d been using up to that point. Primarily this was so I could ride for further since my phone generally died after a couple of hours of constant screen and GPS action. I went for the Garmin 810.

In a sense the 810 was a step back from a phone with its tiny screen and washed out colours, but I grew to love its turn by turn navigation, even if it was also the cause of my major annoyance with it, the occasional crash that only seemed to happen when using the turn by turn navigation.

But recently the USB socket packed in, my computer no longer recognised the Garmin when I plugged it in. Although I could still download ride data via my phone’s Bluetooth connection, I now had no way of uploading new routes to the device, so decided to treat myself with a 830. Those crashes had put me off Garmin devices somewhat but when I compared what was available from other companies, I came to the conclusion the 830 still looked like the best choice.

So this is a mini review of how the world of Garmin has moved on in the last 5 years and whether an upgrade is worth it.

On first sight, the 830 looks fairly similar to the 810 with almost exactly the same dimensions. The buttons have moved from the top to the front of the device which makes it rather difficult to press them if the device is mounted on your stem. Because of this, I’ve switched to using the out front bracket which is thankfully included with the device.

It’s when you turn it on, you start to notice the differences. The screen has a much higher resolution and the colours are much improved over the 810. The software should be familiar to anyone who has used an 810, but there have been many small tweaks and changes. Moving around the screens is very responsive and I’ve had no problems when wearing gloves.

I remember when I bought the 810, only a very basic base map was provided. I could buy a proper map from Garmin or try to figure out how to download OSM maps and install them. I went for OSM maps initially but eventually bought Garmin maps in the vain hope it would fix the crashes I was experiencing. Now the 830 comes with its own maps

Integration with my phone has moved on since the 810. I can now get routes directly on to the Garmin via the Strava app, rather than having to download from Strava and upload to the bike computer. I even get notifications from my phone appearing on the Garmin. Admittedly I’m generally on my bike to get away from these kind of things, so this may not be something I leave enabled, but I can imagine it would be useful when trying to meet up with someone.

I was always a big fan of turn by turn navigation. This has been improved so that when I go off route, the 830 will recalculate the route, even if this often just means telling me to make a U-turn. I even get told about sharp corners ahead. it can get a bit confused if a route covers the same bit of road more than once, trying to send me the wrong way, but it figures it out fairly quickly.

I’ve had the 830 for just over a month and ridden just over 300 miles. In that time I’ve had zero crashes. I wouldn’t like to say the software reliability problems of the past are gone (since crashes seemed to happen more frequently on longer rides which I’ve not really been doing due to the winter months) but I’m feeling more confident than I did with the 810.

I think the only complaint I have about the 830 is a problem I have with most bike computers, the price. I have a Garmin sat-nav for our car which provides pretty much the same functionality as the 830, but for a quarter of the price. There are differences in the software and the 830 integrates with heart rate monitors and power meters, but can such a vast difference in price be justified? I guess if this keeps going for five years, then maybe it can. Either way, I’ve been very impressed after my first month

Land Registry data December 2019

I’ve uploaded the latest property sales data for England and Wales to my website. Nothing of note to report, annual house price inflation is stuck at 1%, sales seem to be holding steady.

Land Registry data November 2019

I’ve uploaded the latest property sales data from the Land Registry for England and Wales to my website. Prices continue to creep higher, number of sales are fairly static

Land Registry sales data October 2019

I’ve uploaded the latest property data to my website. Nothing of interest to report…

ONS postcode data for November 2019

I’ve uploaded the latest ONS postcode data to my website. I’ve run my usual sanity checks but let me know if you find any problems

Land Registry data for September 2019

I’ve uploaded the latest house price data to my website. Prices continue on their sluggish ascent, although the annual change is now below 1%. The number of sales seems surprisingly resilient, I guess whatever madness is going on in the UK, people still need to move house

New server

The server running doogal.co.uk was getting a little long in the tooth and with more data getting added every month, was starting to slow down somewhat. So I’ve finally got round to upgrading to a shiny new server with a SSD, more memory and the latest versions of the software used. Hopefully it is all working as expected, but let me know if you spot any problems

Land Registry data August 2019

My server is straining to import the latest house price data, but not to worry, I’m in the process of building a new server with more memory and a faster hard drive which will be up and running shortly.

The latest data is fairly dull, as prices continue to increase very slowly whilst looking like they may head negative some time soon

Land Registry data for July 2019

Another month, another set of house price and sales data from the Land Registry, which I’ve uploaded to my website. Prices are still managing to rise, but at an increasingly slow pace

UK postcode update

I’ve uploaded the latest postcode update from the ONS to my website. I’ve run my usual sanity checks and everything appears correct but let me know if you spot anything amiss.

Land Registry data for June 2019

I’ve uploaded the latest property sales data for England and Wales to my website. Not much to report, the long slow trend towards house price deflation continues

The random address generator is back

My random addresses generator is back. After having my hand slapped by Google Maps, I’ve had to re-implement it using my own database of property sales. Because of that change, the addresses returned are only from England and Wales, but they will now always be actual addresses rather than the occasional A-Road that used to be returned. And you no longer need to provide a Google Maps API key, yay!

I’ve also removed a few other features which I’ll probably start adding back as time permits. Let me know if there’s anything you desperately need.

Land Registry data for May 2019

I’ve uploaded the latest house price data for England and Wales to my website. The data is so boring at the moment I can’t find anything of interest to say about it. The annual rate of inflation has been stuck at 1.7% for the last 3 months.

I will make the prediction that something interesting will happen if/when we leave the EU but since I don’t want to spend my days clearing out abusive messages from the comments section, I won’t say which way I think prices will go…

Isoline routing

Here Maps has an interesting feature that I’ve not seen in other mapping solutions. Called isoline routing, it answers the question “Starting from point A, how far can I travel in X amount of time?”. I’m not sure how useful it is, but it’s certainly interesting to play with, so I’ve added a page implementing it on my site.

Cycling is not supported by the API so I’m using the walking option adjusted for the faster travelling time, although the results may not be completely accurate.

Let me know if there’s anything you’d like to see added to it.

House price data for April 2019

I’ve uploaded the latest Land Registry data for April 2019 to my website. House price inflation continues to drift lower, to just under 2%. It looks like flats are actually dropping in price

ONS postcode data May 2019

I’ve uploaded the latest UK postcode data to my website. As ever if you spot anything amiss, let me know

Help me raise some money for the British Heart Foundation

On June 16th I’ll be riding from London to Brighton to raise funds for the British Heart Foundation. I’ve ridden RideLondon before and was pleasantly surprised at the donations I received from friends, family and complete strangers then so I’m hoping the same happens again.

If you’ve found this blog or my website useful, or are just feeling generous, then please consider donating some money. I will certainly appreciate it, as will the British Heart Foundation.

Why BHF? Well they organise the event so that’s who gets the money. But from a personal perspective, heart disease runs in my family so it’s close to my, er, heart…

Land Registry data for March 2019

I’ve uploaded the latest house price data for England and Wales to my website. Number of sales are fairly static with house price inflation dropping slightly to just under 2%. At the regional level, ignoring the ever volatile central London postcodes (due to a small number of high value transactions), Luton is currently the place with the fastest rising prices and Southall is the place with the fastest dropping prices.

Converting KML maps from Google Maps to Here Maps

When Google went insane and decided to charge excessive amounts for use of their mapping APIs I looked for an alternative. One of the features I needed was support for KML, since my website uses it quite extensively. Which led me to Here Maps.

After a fair amount of work, I managed to convert most of my pages to use Here Maps, but there’s still a few stuck using Google Maps due to features that are unique to Google Maps. This was OK, since my usage was now mostly under the $200 per month of free credit. But recently one of my Google Maps pages started to get a lot of hits due to a new inbound link and I zoomed past $200 free credit into eye-wateringly expensive territory. So time to convert that page to Here Maps.

Converting maps that load KML from Google Maps to Here Maps is generally straightforward, just learn a different API and redo the JavaScript on your page. But the architecture of KML support on the two platforms is different. Google loads the KML on their servers, generates map tiles and uses those to display the KML file. Here Maps loads the KML file in the browser and add markers, polylines etc to the map directly.

The Google approach has one major advantage, it copes well with large KML files. Since the move to Here Maps, I’ve had to stop loading up KML files that I know have more than a few thousand markers in them. Google’s KML support also means you can load up KML files from external sources, whereas Here Maps will generally fail with external KML (unless CORS has been configured to support it on the other server).

Google KML has a few disadvantages. If the KML changes regularly then you’ll probably suffer from caching issues, since the old map tiles can keep getting returned for some time after the KML changes. Also the rendering isn’t as good as Here Maps, since the KML is rendered as an image at each zoom level rather than as live objects on the map

The thing that had stopped me moving this page over to Here Maps was the inability to display remote KML data. Then it struck me that the fix for that was fairly straightforward. Add a local piece of server code that loads up the KML file from the remote source and returns it to the browser so it’s treated as a local URL. That was easy enough to code up. Then I just needed to cope with a few edge cases, Google Maps copes with KMZ files, but Here Maps doesn’t. And some servers didn’t like requests coming from something that wasn’t a browser. So I eventually came up with this

public void ProcessRequest(HttpContext context) {
 context.Response.ContentType = "application/vnd.google-earth.kml+xml";
 var url = context.Request.QueryString["url"];
 var httpRequest = (HttpWebRequest) WebRequest.Create(url);
 httpRequest.Method = "GET";
 // pretend to be a browser
 httpRequest.UserAgent =
  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";

 using(var httpResponse = (HttpWebResponse) httpRequest.GetResponse()) {
  var responseStream = httpResponse.GetResponseStream();

  if (responseStream != null) {
   var archiveStream = new MemoryStream();
   responseStream.CopyTo(archiveStream);
   archiveStream.Position = 0;

   // see if it's a zip file
   try {
    var archive = new ZipArchive(archiveStream);
    using(var stream = archive.Entries[0].Open())
    using(var archiveReader = new StreamReader(stream)) {
     context.Response.Write(archiveReader.ReadToEnd());
    }
   } catch (Exception) {
    archiveStream.Position = 0;
    var reader = new StreamReader(archiveStream);
    var response = reader.ReadToEnd();
    context.Response.Write(response);
    reader.Close();
   }

   // Close both streams.

   responseStream.Close();
  }
 }
}

Updated postcode boundaries

The boundaries for postcode areas, districts and sectors were rather out of date on my website. I originally downloaded them from a website that no longer exists. After searching around the web for updated files, I came up empty handed. The ones I found were either just as out of date as mine or had restrictive licensing attached to them.

So I set about building my own, which are now live on the site.

You can stop reading now unless you have an interest in the technical details of how I built the boundary files.

Since the freely available postcode data doesn’t include postcode unit boundaries, just the centre point of each postcode, the first step was to generate approximate boundaries for postcode units. This can be achieved using a Voronoi diagram, which generates polygons around points so that every location in the polygon is closer to the point than any other points. Although the polygons for each postcode unit are not particularly useful or accurate, combining them together at the district and sector level should give a decent approximation of the actual area covered.

The GIS tool QGIS can calculate Voronoi diagrams so I imported the postcode data into SQL Server and ran the Voronoi tool in QGIS. After several hours it hadn’t produced anything and didn’t give me any indication of when it would complete (it was stuck at 99% complete for most of that time).

So I decided to search for some code that could produce the Voronoi diagram so I could run that and would at least have an idea of where it was in the process. I found this bit of C# code that mostly worked. The output from it was edges rather than polygons, so some further work was required to construct polygons from the edges. Once the polygons were created I put those into SQL Server (adding District and Sector fields for later use) so I could work on them further in QGIS.

The next issue was that the polygons created extended far beyond the shores of the UK. This required the use of the ‘Clip vector by mask layer’ tool in QGIS, using a UK border polygon layer as the mask (note to self for next time I do this – this takes hours to complete). Then the postcode units needed to be merged into districts and sectors. The ‘Dissolve’ tool in QGIS can be used for this, using the District and Sector fields as the dissolve field.

I then had two KML files, one with all postcode districts and one with all postcode sectors. I then wrote some C# code to split the files apart so the individual districts and sectors can be displayed and downloaded.

Land Registry data for February 2019

I’ve just uploaded the latest house price data to my website. Not much to report (2% annual inflation still), but I do have some prettier charts

Land Registry data for January 2019

I’ve uploaded the latest property sales data from the Land Registry to my website. Not much to report, house price inflation remains at 2%, transactions remain flat

UK postcode data for February 2019

I have uploaded the latest UK postcode data from the ONS to my website. I’ve run my usual sanity checks and everything looks OK but let me know if anything appears amiss. Enjoy all the 2,611,698 postcodes (alive and dead)

UK station usage 2018

I’ve added 2018 station usage date for UK stations to my website. Enjoy

Land Registry data for Dec 2018

I’ve uploaded the latest house price data from the Land Registry to my website. Currently every month feels like Groundhog Day, prices continue to rise at about 2%, the number of transactions is stable.

Land Registry house price data November 2018

The good people at the Land Registry were probably taking a well earned vacation, so the data arrived a little later than normal but house price data for England and Wales for November 2018 is now on my site. Not much to report, annual inflation is still sitting at about 2%, the number of transactions are holding steady. Maybe things will get interesting in March, I hear something might be happening around then.

Free for all

As mentioned in a previous post, Google have started charging me for some more Google Maps related things but have also given me some free credit to tide me over. In fact, they think my future usage of Google Maps is going to be so high, they have given me an absolutely huge amount of credits. Since I can’t actually do anything with those credits other than spend them on the Google Maps API, I have removed the requirement for a Google Maps API key on the pages that previously required one. This will remain in place until either my credits run out or we’re close to the end of January. So please go crazy. Happy Christmas.

Important note – if the API key field contains your API key, your API key will still be used!

Land Registry house price data for October 2018

The latest house price data from the Land Registry is now on my site. House prices continue to rise slowly, although trending lower currently.

UK postcode data November 2018

I’ve updated my website with the latest ONS postcode data for November 2018. As ever if you spot anything amiss, let me know

More Google Maps annoyances

The other day I got another email from Google

Hi,

In June 2016, we announced a change to Maps JavaScript API requests. At that time, we gave you temporary free usage based on your consumption to ensure that your applications would continue to function. The services included in this transition period were: Elevation, Directions, Distance Matrix, Geocoding and Places.

We appreciate you as a loyal and long-standing customer. Our goal is to make sure everyone is on a simple, consistent, and scalable plan with Google Maps Platform.Starting on November 29, 2018, we will bill all your usage for Elevation, Directions, Distance Matrix, Geocoding and Places, according to our new pricing plan.

To help you with this transition, we will provide you with two months of credits, which we will automatically apply to your billing account. Please read our FAQs to understand what these credits cover and how to estimate your monthly bill.

Thank you for using Google Maps Platform.

The Google of 2016 was clearly a different company to the Google of 2018, since grandfathering of old customers when a radical change to pricing is introduced is exactly the right thing to do unlike the recent shenanigans. And generally grandfathering is a permanent thing…

So once again I’m going to have to spend some time switching things off, moving other stuff over to one of Google’s many competitors who have more sane pricing and making other things require a Google Maps API key (the route elevation page now does, sorry)

House price data for England and Wales September 2018

I’ve uploaded the latest Land Registry data to my website. Prices continue to creep up at around 2% a year, although several regions have seen falling prices

Implementing my own version of the Google Maps Timezone API

I noticed the other day that my usage of the Google Maps Timezone API was failing. I realised this was down to me not passing in an API key with the call. In their attempts to monetise their Maps API, Google now requires the API key and each call is chargeable. So I added the key and it still didn’t work, although with a different error message. Apparently using a key with a HTTP referrer restriction wasn’t allowed.

So I decided to add a server-side handler on my server that called out to the Timezone API using a server key instead. This was fairly straightforward since it just bounced the AJAX request from the browser to the Timezone API URL.

I checked back the next day to see what my API usage looked like. I’d spent $5 in 24 hours. Continuing with that meant with my other Maps API usage I’d hit the $200 per month free limit and would have to start paying Google money again, something I’ve been loath to do since their ridiculous price increases*

I realised at that point that the Timezone API wasn’t actually doing a huge amount behind the scenes. I guessed there would be libraries out there that could do the same thing but without paying for the privilege. Turns out there is. GeoTimeZone will give the time zone ID for a location and TimeZoneConverter will convert that to a Windows TimeZoneInfo that gives me everything else I needed to build my own version of the Timezone API. The code to do that is something like this

     var lat = double.Parse(HttpContext.Current.Request.QueryString["lat"]);
     var lng = double.Parse(HttpContext.Current.Request.QueryString["lng"]);
     var tz = TimeZoneLookup.GetTimeZone(lat, lng).Result;
// get other info
var tzi = TZConvert.GetTimeZoneInfo(tz);
     // write out as JSON
     var jsonObj = new JObject();
     var rawOffset = tzi.BaseUtcOffset.TotalSeconds;
     jsonObj["dstOffset"] = tzi.GetUtcOffset(DateTime.UtcNow).TotalSeconds - rawOffset;
     jsonObj["rawOffset"] = rawOffset;
     jsonObj["timeZoneId"] = tz;
     jsonObj["timeZoneName"] = tzi.StandardName;
     jsonObj["status"] = "OK";
    var json = JsonConvert.SerializeObject(jsonObj);
     HttpContext.Current.Response.Write(json);

The only thing to consider is that time zones change so it’s worth keeping the two packages up to date.

* For the record, I used to pay about $200 a month to Google. Now I pay about the same to here maps and nothing to Google. I’m intrigued to know how their new pricing has worked out for them, I’m assuming most websites would have made the same decision I did and moved somewhere else.

Downloads on Chrome

If you’re having problems downloading files from my website, it may be down to the latest release of Chrome. It seems it’s got a bug that stops downloads working in some cases. I’ve found a workaround for some downloads but there are a few places where I haven’t figured out what to do you yet. Right clicking and selecting ‘Save link as..’ may fix the issue but if that’s not offered as an option, all I can suggest is switching to a different browser

New features on doogal.co.uk

I’ve been making a few enhancements to the website recently so thought I’d point out a few things that might be of interest.

Editable labels on geocoded locations – The labels displayed on the map after geocoding can now be edited. Editing the label will update the text output and the KML output.

Coloured elevation charts – I’ve updated my elevation charts to include colours based on the gradient at that point of the route/segment. Here’s Box Hill in all its technicolour glory. Due to the variable quality of elevation data, it can sometimes be difficult to calculate sensible gradients, so let me know if you see anything that doesn’t look sensible.

View your starred segments – The segment viewer page had fairly limited functionality before so I’ve now added the ability to view all your starred segments. If you’ve connected my website to Strava on the segment explorer page, you should see a ‘View starred segments’ button which will display them all on a map. This will be quite slow to load initially if the segments are not already available on the site, but should be quicker on subsequent loads. It’s also currently limited to 100 segments.

House price data for August 2018

Another month has flown by and it’s time to upload some more Land Registry data to my website. Annual house price inflation sits at 2%, another drop that may be becoming a trend. Number of sales also appears to be on a downward trend

House price data for July 2018

I’ve uploaded the latest house price data from the Land Registry to my website. Not much excitement in the overall figures but there’s more interesting data at the regional level. A year after being the City of Culture, Hull is suffering the hangover and the wild swings continue in central London (although this is mostly down to the small number of sales that occur in the WC postcode area).

Decoding paths in here maps

In my panic to move my website off Google Maps, I took some shortcuts. One of those was to continue to use the helper function google.maps.geometry.encoding.decodePath. I primarily use this for decoding the paths of Strava segments, since these are encoded using the same algorithm as Google. But loading up the Google Maps API script for a single function is rather ridiculous, so I’ve converted the path decoder from the rather marvellous (and seemingly abandoned) Strava.NET library.

For your pleasure, here is some TypeScript that will create a here maps LineString from an encoded path. Other map providers are available and adjusting it for them should be fairly straightforward.

function decodePath(polylinechars: string): H.geo.LineString {
  var poly = new H.geo.LineString();

  if (polylinechars == null || polylinechars === "") {
    return poly;
  }

  var index = 0;

  var currentLat = 0;
  var currentLng = 0;

  while (index < polylinechars.length) {
    // calculate next latitude
    var sum = 0;
    var shifter = 0;
    var next5bits;
    do {
      next5bits = polylinechars.charCodeAt(index++) - 63;
      sum |= (next5bits & 31) << shifter;
      shifter += 5;
    } while (next5bits >= 32 && index < polylinechars.length);

    if (index >= polylinechars.length)
      break;

    currentLat += (sum & 1) == 1 ? ~(sum >> 1) : (sum >> 1);

    // calculate next longitude
    sum = 0;
    shifter = 0;
    do {
      next5bits = polylinechars.charCodeAt(index++) - 63;
      sum |= (next5bits & 31) << shifter;
      shifter += 5;
    } while (next5bits >= 32 && index < polylinechars.length);

    if (index >= polylinechars.length && next5bits >= 32)
      break;

    currentLng += (sum & 1) == 1 ? ~(sum >> 1) : (sum >> 1);

    poly.pushPoint(new H.geo.Point(currentLat / 100000.0, currentLng / 100000.0));
  }

  return poly;
}

UK postcode date August 2018

I’ve uploaded the latest postcode data from the ONS to my website. I’ve checked the data with my usual sanity check but let me know if you spot anything amiss

Here Maps full screen control

One thing missing from Here Maps is a full screen control, but fortunately it’s possible to add your own custom controls to the map. This isn’t brilliantly documented but a support person from Here Maps was able to help me out with an example. That combined with the fact I’d built something similar in Google Maps before it had its own full screen control. So here’s a TypeScript function that will add a full screen button to a Here Map.

function hereMapsFullScreenControl(ui: H.ui.UI) {
  // add custom UI control
  const myCustomControl = new H.ui.Control();
  myCustomControl.addClass("customControl");
  ui.addControl("myCustomControl", myCustomControl);
  // Set the position of the control in the UI's layout
  myCustomControl.setAlignment(H.ui.LayoutAlignment.TOP_RIGHT);

  const myCustomPanel = new H.ui.base.OverlayPanel();
  myCustomPanel.addClass("customPanel");
  myCustomControl.addChild(myCustomPanel);

  // store original styles
  const mapDiv = ui.getMap().getElement() as HTMLElement;
  let divStyle: CSSStyleDeclaration = mapDiv.style;
  if (mapDiv.runtimeStyle) {
    divStyle = mapDiv.runtimeStyle;
  }
  const originalPos: string = divStyle.position;
  let originalWidth: string = divStyle.width;
  let originalHeight: string = divStyle.height;
  // ie8 hack
  if (originalWidth === "") {
    originalWidth = mapDiv.style.width;
  }
  if (originalHeight === "") {
    originalHeight = mapDiv.style.height;
  }
  const originalTop: string = divStyle.top;
  const originalLeft: string = divStyle.left;
  const originalZIndex: string = divStyle.zIndex;
  let bodyStyle: CSSStyleDeclaration = document.body.style;
  if (document.body.runtimeStyle) {
    bodyStyle = document.body.runtimeStyle;
  }
  const originalOverflow: string = bodyStyle.overflow;

  const myCustomButton = new H.ui.base.PushButton({
    label: "Full screen",
    onStateChange: (evt) => {
      // OK, button state changed... if it's currently down
      if (myCustomButton.getState() === H.ui.base.Button.State.DOWN) {
        // go full screen
        mapDiv.style.position = "fixed";
        mapDiv.style.width = "100%";
        mapDiv.style.height = "100%";
        mapDiv.style.top = "0";
        mapDiv.style.left = "0";
        mapDiv.style.zIndex = "100";
        document.body.style.overflow = "hidden";
      } else {
        // exit full screen
        if (originalPos === "") {
          mapDiv.style.position = "relative";
        } else {
          mapDiv.style.position = originalPos;
        }
        mapDiv.style.width = originalWidth;
        mapDiv.style.height = originalHeight;
        mapDiv.style.top = originalTop;
        mapDiv.style.left = originalLeft;
        mapDiv.style.zIndex = originalZIndex;
        document.body.style.overflow = originalOverflow;
      }
      ui.getMap().getViewPort().resize();
    },
  });

  myCustomPanel.addChild(myCustomButton as any);
  myCustomPanel.setState(H.ui.base.OverlayPanel.State.OPEN);
}

Land Registry data for June 2018

The server is still churning through the data for the England and Wales house price data for June 2018 but the top-level data is there. Annual inflation has dropped a little, but not enough to warrant any excitement.

The economics of the new Google Maps API pricing

The insane new pricing for the Google Maps API with a $200 per month credit leads to an odd situation. For anyone using less than $200 a month, Google Maps is probably the best option (although other map providers also generally have some kind of free tier). For anyone going over that limit, you’d have to be insane to use Google Maps since it now costs so much more than other map providers.

So if you’re in a similar situation to me and you have a bunch of pages using Google Maps, the sensible option is to convert them until you get to a stage where your Google costs are approximately $6.67 per day. The Billings Report in the Google APIs console gives a fairly up to date picture of what’s going on (turn off the ‘Include credit in costs’ option). My current costs are about $20 per day so some work still to be done.

I’d much prefer to be improving the site, but since Google have decided to be dicks, I have to rewrite stuff until I’m no longer paying them a penny…

House price data May 2018

The Land Registry data for May 2018 is now on my website. House prices continue to fail to do anything exciting, continuing their slow ascent upwards.

A reply to Google Maps

I received an email from Google Maps with a subject line of ‘Action needed: Contact Google Maps Platform for volume pricing’, as follows. I thought I’d translate it and reply since it came from a no-reply email address.

Hi,

We are following up on our most recent announcement. This is a reminder that Google Maps Platform’s new terms and pricing will go into effect July 16.

Translation – we are massively increasing our prices on July 16

The 2 months of credit we extended to you will automatically apply to your billing account on that date.

Translation – this won’t cover the increase in prices

You are eligible for a significantly discounted price on your monthly bill, based on your usage over the last three months.

Translation – the discounted price is not that discounted and will still be more expensive than our competitors

If you have not yet contacted us, we strongly recommend that you contact us to learn more about our volume pricing and how it can benefit you.

Reply – I did contact you. Your support staff seem kind of stressed. They sent me off to one of your partners who was only able to offer me a deal that would mean all of my advertising revenue went to Google Maps

Thank you for using Google Maps Platform.

It was fun while it lasted but I’m busily moving all my maps to another provider with more sane pricing. I would really love to know the rationale behind this move because it makes zero sense to me.

Reducing Google Maps costs

I’m sure I’m not the only person who will be hit hard by the huge increase in the cost of Google Maps. I’ve been using Google Maps for years on my site and it’s been absolutely brilliant. But my monthly costs will increase from a couple of hundred pounds to several thousand pounds, which will wipe out any money generated through advertising. And since I was given very little notice of the increase in price, I’ve had to move quickly to try to keep my site useful but still economical. So here’s some suggestions if you’re in the same boat.

Use embedded maps – these are still free and if you’re showing a simple map with a marker then they might meet your needs. You lose Street View but as a quick temporary fix, this seems the most straightforward option.

Don’t load things immediately – I’ve been in the habit of loading maps and showing information from other Google Maps APIs as soon as the page loads, because it hardly cost anything. I’m now looking very closely at what is key information and what may only be useful to some users. If it’s only going to be useful to some users, I’m adding a button to display that data.

Turn things off – sometimes I’ve added stuff just because it looked like fun without considering whether it would be useful at all. Just removing it is generally easier than the previous option.

Other APIs – At the time of writing, my Google APIs console doesn’t even tell me the usage of some APIs that will soon cost a lot of money to use (Places and autocomplete for example). the only suggestion here is to keep an eye on that console because hopefully usage data will appear some time before we start getting charged

Move to a different map provider – If the prices of other map providers remains the same, then I’m not sure why anyone would choose Google Maps anymore. Their prices are completely out of whack with the rest of the world. But maybe they are just the first company to decide they need to charge more because actually it costs a lot to provide a mapping API. If that is the case, the other map providers will probably breathe a sigh of relief and up their prices to a similar level. And if that is the case, moving to another company could be a time-consuming and ultimately fruitless endeavour. That said, the way Google has handled this price rise has made me lose confidence in them so I’m looking at alternatives. I’ve converted one map to Here Maps fairly easily (I chose them mainly because they support KML) and will convert more as time permits

Goodbye Google Maps – important notice

A while back I received an email from Google Maps telling me they were introducing new pricing, but the email included the line “Based on your project usage over the last 3 months and our new pricing plan, we estimate that your new cost will be less than $200 a month and will be covered by our $200 monthly free credit” so I was pretty chuffed since I currently spend about £100 a month on Google Maps. I then promptly forgot about it.

But I then received another email saying the changes were being postponed and I thought it was time to look at how much I’d be paying. After running a few numbers, I realised I’ll have to pay several thousand pounds a month to continue to use Google Maps. Since this will wipe out any advertising revenue I make (via Google of course…), it looks like I’ll have to start removing and replacing various parts of the site.

The Google API Console is fairly useless. Although it tells me my API usage, it doesn’t tell me which pages are causing the usage so I don’t have a clear insight into what I need to remove. But two pages that are likely to be affected are the batch geocoding and batch reverse geocoding pages since they certainly generate a lot of requests. Other pages likely to be affected are random addresses, route elevation and driving distance. Other pages may fail to load maps or fail in other ways after July 16th. Apologies in advance

UK postcode data for May 2018

I’ve uploaded the latest UK postcode data to my website. I’ve done my usual sanity checks but if you spot anything amiss, let me know

Land Registry house price data April 2018

I’ve uploaded the latest Land Registry house price data to my website. I’d read stories of annual house price falls but the Land Registry data is showing a 3% annual increase. The Land Registry numbers are a few months behind other house price indices so a fall may be on its way in a few months

One month later and TSB online banking is still a disaster

One month ago TSB tried to migrate their customers to a new online banking platform. Things didn’t go particularly well and although the media have lost interest in the problems, as a TSB customer and IT geek I’m still interested in what’s happening and thought I’d touch on a few of the issues.

The most glaring issue is that my mortgage account still doesn’t appear in my list of accounts. I assume I still have a mortgage since the payments are still being taken from my current account but there has been no sighting of it since the ‘upgrade’.

A few days ago, I tried to login, only to see an error message, <div>Sorry, we are currently experiencing technical issues. Please try again later.</div>. Not being able to login isn’t great but not totally unexpected given the recent troubles. Displaying HTML tags in your error messages is a fairly minor issue but suggests a lack of attention to detail that can lead to more serious issues down the road.

Now lets go under the hood by firing up Dev Tools in Chrome. The first error I saw was complaining about Symantec certificates which Chrome will stop supporting in a future release. Let’s hope someone at TSB renews that fairly soon before things break properly.

But the next set of errors are slightly more concerning, WebSocket connection to 'wss://127.0.0.1:3389/' failed, something on the web page is trying to connect to a server on my local machine! I can only assume this is some kind of debugging code but should definitely not be getting pushed out to their production system.

Next we have a failure to load a locale file for US English, which is odd since I’m set up for UK English. Again, nothing serious but shows a lack of attention to detail.

Finally, let’s run the page through Lighthouse, the Chrome web page checker. The performance score is a mighty 18 out of 100. This seems mostly due to Javascript and CSS that isn’t bundled together and isn’t being loaded using async or defer, meaning everything has to load before the page can be displayed.

And that’s what I found spending a few minutes on the TSB website, who knows what could be found if I was an evil hacker wanting to steal some money

Converting KML file to shapefiles

For a long time I’ve been searching for some code that I can use on my website to convert the various KML postcode files that are available. Since many of these files are generated on the fly, I didn’t want to use a one off convertor. But I’ve always failed to find anything that will do the job. Since I don’t know the internals of shapefiles, it’s not something I wanted to write myself.

So I’ve kind of given up, but if you need to convert any of the KML files yourself, then this tool may be useful. From my admittedly not very thorough testing, it seems to do a good job of the conversion

Land Registry house price data March 2018

I’ve uploaded the latest property data for England and Wales from the Land Registry to my website. Annual house price inflation has been slowly creeping up and now stands at 3%. Sales data is harder to get a handle on since it can take months for all the sales data to come through but it looks like sale may be dropping off a little.