Saturday, November 21, 2020

Postcode data for November 2020

I’ve uploaded the latest UK postcode data from the ONS to my website. I have checked that it looks OK but I often miss something so let me know if you spot anything amiss

Thursday, October 29, 2020

England and Wales house price data update

The latest update from the Land Registry for September 2020 is now live on my site. Data still seems to be slow in coming through so the average sale price and number of sales is still volatile. I’m guessing the numbers from the summer are now complete which show prices flatlining and sales volumes much lower than normal

Tuesday, September 29, 2020

Land Registry data for August 2020

The latest property data for England and Wales is now on my website. The number of sales has certainly fallen over the last few months, even though I’ve been hearing of a property boom. And I’m not seeing much evidence of prices rising much either

Friday, August 28, 2020

Land Registry data for July 2020

The latest house price data from the Land Registry is now on my website. Sales are still way down but it’s too early to see what affect the Stamp Duty holiday has had, I’d imagine sales will be up over the coming months

Saturday, August 22, 2020

Wednesday, July 29, 2020

House price data June 2020

I’ve uploaded the Land Registry data for June 2020 to my website. I reported a couple of months ago that there had been 7 sales in April. This has now been revised up to over 20,000 sales. Whilst not as terrible as the initial figures suggested, that’s still much lower activity than we’ve been seeing over the last few years. And although there’s been a marked drop in the prices paid, those prices haven’t affected the annual averages yet.

Wednesday, July 01, 2020

Property sales May 2020

I’ve uploaded the latest Land Registry property data to my site. There’s a continued lack of sales being completed, although last month’s figures have been revised upwards from 7 to 279 sales. That is still dramatically lower than the usual number of sales

If you’re wondering why the current average sale price has dropped significantly but the annual change in price has increased, this is due to the tiny number of sales happening, the sales dropping out of the other end are making a much bigger difference to the stats.

Friday, June 05, 2020

UK postcode data for May 2020

The ONS have published their latest postcode data which is now uploaded to my site. As ever, I’ve run some sanity checks (codes for administrative areas have changed yet again!) and everything looks to be correct. Let me know if you see anything amiss

Tuesday, June 02, 2020

(lack of) property sales April 2020

What happens to property sales in the middle of a pandemic? Perhaps unsurprisingly they fall off a cliff. According to the Land Registry, there were 7 sales in England and Wales in April, although that will probably go up a little as the data comes in for future months. With such a small number of sales, the price statistics are fairly meaningless

Friday, May 01, 2020

Land Registry data for March 2020

In the before days, people bought and sold houses. I’ve uploaded the data for England and Wales from March to my website. Since the data is never complete for the previous month, it’s difficult to see at this point the effect of the lockdown, but I assume it will become more obvious next month.

Monday, March 30, 2020

Land Registry data for February 2020

It may feel like a lifetime ago, but in the not too distant past people were able to buy and sell houses. I’ve uploaded the latest Land Registry data to my website which is fairly uninteresting, with prices still ticking up slowly. I imagine next month’s data will look very different.

Saturday, February 29, 2020

Property data for January 2020

I’ve uploaded the latest Land Registry data for England and Wales to my website, covering sales up to January 2020. Prices continue to creep up, number of sales are hovering around the level they’ve been at since 2013 (when Help To Buy was introduced)

Thursday, February 20, 2020

Monday, February 03, 2020

Moving from a Garmin 810 to a Garmin 830

About 5 years ago, I decided I needed to buy myself a dedicated bike computer, rather than the phone I’d been using up to that point. Primarily this was so I could ride for further since my phone generally died after a couple of hours of constant screen and GPS action. I went for the Garmin 810.

In a sense the 810 was a step back from a phone with its tiny screen and washed out colours, but I grew to love its turn by turn navigation, even if it was also the cause of my major annoyance with it, the occasional crash that only seemed to happen when using the turn by turn navigation.

But recently the USB socket packed in, my computer no longer recognised the Garmin when I plugged it in. Although I could still download ride data via my phone’s Bluetooth connection, I now had no way of uploading new routes to the device, so decided to treat myself with a 830. Those crashes had put me off Garmin devices somewhat but when I compared what was available from other companies, I came to the conclusion the 830 still looked like the best choice.

So this is a mini review of how the world of Garmin has moved on in the last 5 years and whether an upgrade is worth it.

On first sight, the 830 looks fairly similar to the 810 with almost exactly the same dimensions. The buttons have moved from the top to the front of the device which makes it rather difficult to press them if the device is mounted on your stem. Because of this, I’ve switched to using the out front bracket which is thankfully included with the device.

It’s when you turn it on, you start to notice the differences. The screen has a much higher resolution and the colours are much improved over the 810. The software should be familiar to anyone who has used an 810, but there have been many small tweaks and changes. Moving around the screens is very responsive and I’ve had no problems when wearing gloves.

I remember when I bought the 810, only a very basic base map was provided. I could buy a proper map from Garmin or try to figure out how to download OSM maps and install them. I went for OSM maps initially but eventually bought Garmin maps in the vain hope it would fix the crashes I was experiencing. Now the 830 comes with its own maps

Integration with my phone has moved on since the 810. I can now get routes directly on to the Garmin via the Strava app, rather than having to download from Strava and upload to the bike computer. I even get notifications from my phone appearing on the Garmin. Admittedly I’m generally on my bike to get away from these kind of things, so this may not be something I leave enabled, but I can imagine it would be useful when trying to meet up with someone.

I was always a big fan of turn by turn navigation. This has been improved so that when I go off route, the 830 will recalculate the route, even if this often just means telling me to make a U-turn. I even get told about sharp corners ahead. it can get a bit confused if a route covers the same bit of road more than once, trying to send me the wrong way, but it figures it out fairly quickly.

I’ve had the 830 for just over a month and ridden just over 300 miles. In that time I’ve had zero crashes. I wouldn’t like to say the software reliability problems of the past are gone (since crashes seemed to happen more frequently on longer rides which I’ve not really been doing due to the winter months) but I’m feeling more confident than I did with the 810.

I think the only complaint I have about the 830 is a problem I have with most bike computers, the price. I have a Garmin sat-nav for our car which provides pretty much the same functionality as the 830, but for a quarter of the price. There are differences in the software and the 830 integrates with heart rate monitors and power meters, but can such a vast difference in price be justified? I guess if this keeps going for five years, then maybe it can. Either way, I’ve been very impressed after my first month

Saturday, February 01, 2020

Land Registry data December 2019

I’ve uploaded the latest property sales data for England and Wales to my website. Nothing of note to report, annual house price inflation is stuck at 1%, sales seem to be holding steady.

Sunday, November 24, 2019

Saturday, November 02, 2019

Land Registry data for September 2019

I’ve uploaded the latest house price data to my website. Prices continue on their sluggish ascent, although the annual change is now below 1%. The number of sales seems surprisingly resilient, I guess whatever madness is going on in the UK, people still need to move house

Thursday, October 03, 2019

New server

The server running doogal.co.uk was getting a little long in the tooth and with more data getting added every month, was starting to slow down somewhat. So I’ve finally got round to upgrading to a shiny new server with a SSD, more memory and the latest versions of the software used. Hopefully it is all working as expected, but let me know if you spot any problems

Monday, September 30, 2019

Land Registry data August 2019

My server is straining to import the latest house price data, but not to worry, I’m in the process of building a new server with more memory and a faster hard drive which will be up and running shortly.

The latest data is fairly dull, as prices continue to increase very slowly whilst looking like they may head negative some time soon

Saturday, August 31, 2019

Land Registry data for July 2019

Another month, another set of house price and sales data from the Land Registry, which I’ve uploaded to my website. Prices are still managing to rise, but at an increasingly slow pace

Sunday, August 25, 2019

UK postcode update

I’ve uploaded the latest postcode update from the ONS to my website. I’ve run my usual sanity checks and everything appears correct but let me know if you spot anything amiss.

Saturday, July 06, 2019

The random address generator is back

My random addresses generator is back. After having my hand slapped by Google Maps, I’ve had to re-implement it using my own database of property sales. Because of that change, the addresses returned are only from England and Wales, but they will now always be actual addresses rather than the occasional A-Road that used to be returned. And you no longer need to provide a Google Maps API key, yay!

I’ve also removed a few other features which I’ll probably start adding back as time permits. Let me know if there’s anything you desperately need.

Friday, June 28, 2019

Land Registry data for May 2019

I’ve uploaded the latest house price data for England and Wales to my website. The data is so boring at the moment I can’t find anything of interest to say about it. The annual rate of inflation has been stuck at 1.7% for the last 3 months.

I will make the prediction that something interesting will happen if/when we leave the EU but since I don’t want to spend my days clearing out abusive messages from the comments section, I won’t say which way I think prices will go…

Sunday, June 23, 2019

Isoline routing

Here Maps has an interesting feature that I’ve not seen in other mapping solutions. Called isoline routing, it answers the question “Starting from point A, how far can I travel in X amount of time?”. I’m not sure how useful it is, but it’s certainly interesting to play with, so I’ve added a page implementing it on my site.

Cycling is not supported by the API so I’m using the walking option adjusted for the faster travelling time, although the results may not be completely accurate.

Let me know if there’s anything you’d like to see added to it.

Friday, May 31, 2019

House price data for April 2019

I’ve uploaded the latest Land Registry data for April 2019 to my website. House price inflation continues to drift lower, to just under 2%. It looks like flats are actually dropping in price

Sunday, May 05, 2019

Help me raise some money for the British Heart Foundation

On June 16th I’ll be riding from London to Brighton to raise funds for the British Heart Foundation. I’ve ridden RideLondon before and was pleasantly surprised at the donations I received from friends, family and complete strangers then so I’m hoping the same happens again.

If you’ve found this blog or my website useful, or are just feeling generous, then please consider donating some money. I will certainly appreciate it, as will the British Heart Foundation.

Why BHF? Well they organise the event so that’s who gets the money. But from a personal perspective, heart disease runs in my family so it’s close to my, er, heart…

Wednesday, May 01, 2019

Land Registry data for March 2019

I’ve uploaded the latest house price data for England and Wales to my website. Number of sales are fairly static with house price inflation dropping slightly to just under 2%. At the regional level, ignoring the ever volatile central London postcodes (due to a small number of high value transactions), Luton is currently the place with the fastest rising prices and Southall is the place with the fastest dropping prices.

Saturday, April 13, 2019

Converting KML maps from Google Maps to Here Maps

When Google went insane and decided to charge excessive amounts for use of their mapping APIs I looked for an alternative. One of the features I needed was support for KML, since my website uses it quite extensively. Which led me to Here Maps.

After a fair amount of work, I managed to convert most of my pages to use Here Maps, but there’s still a few stuck using Google Maps due to features that are unique to Google Maps. This was OK, since my usage was now mostly under the $200 per month of free credit. But recently one of my Google Maps pages started to get a lot of hits due to a new inbound link and I zoomed past $200 free credit into eye-wateringly expensive territory. So time to convert that page to Here Maps.

Converting maps that load KML from Google Maps to Here Maps is generally straightforward, just learn a different API and redo the JavaScript on your page. But the architecture of KML support on the two platforms is different. Google loads the KML on their servers, generates map tiles and uses those to display the KML file. Here Maps loads the KML file in the browser and add markers, polylines etc to the map directly.

The Google approach has one major advantage, it copes well with large KML files. Since the move to Here Maps, I’ve had to stop loading up KML files that I know have more than a few thousand markers in them. Google’s KML support also means you can load up KML files from external sources, whereas Here Maps will generally fail with external KML (unless CORS has been configured to support it on the other server).

Google KML has a few disadvantages. If the KML changes regularly then you’ll probably suffer from caching issues, since the old map tiles can keep getting returned for some time after the KML changes. Also the rendering isn’t as good as Here Maps, since the KML is rendered as an image at each zoom level rather than as live objects on the map

The thing that had stopped me moving this page over to Here Maps was the inability to display remote KML data. Then it struck me that the fix for that was fairly straightforward. Add a local piece of server code that loads up the KML file from the remote source and returns it to the browser so it’s treated as a local URL. That was easy enough to code up. Then I just needed to cope with a few edge cases, Google Maps copes with KMZ files, but Here Maps doesn’t. And some servers didn’t like requests coming from something that wasn’t a browser. So I eventually came up with this


public void ProcessRequest(HttpContext context) {
 context.Response.ContentType = "application/vnd.google-earth.kml+xml";
 var url = context.Request.QueryString["url"];
 var httpRequest = (HttpWebRequest) WebRequest.Create(url);
 httpRequest.Method = "GET";
 // pretend to be a browser
 httpRequest.UserAgent =
  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";

 using(var httpResponse = (HttpWebResponse) httpRequest.GetResponse()) {
  var responseStream = httpResponse.GetResponseStream();

  if (responseStream != null) {
   var archiveStream = new MemoryStream();
   responseStream.CopyTo(archiveStream);
   archiveStream.Position = 0;

   // see if it's a zip file
   try {
    var archive = new ZipArchive(archiveStream);
    using(var stream = archive.Entries[0].Open())
    using(var archiveReader = new StreamReader(stream)) {
     context.Response.Write(archiveReader.ReadToEnd());
    }
   } catch (Exception) {
    archiveStream.Position = 0;
    var reader = new StreamReader(archiveStream);
    var response = reader.ReadToEnd();
    context.Response.Write(response);
    reader.Close();
   }

   // Close both streams.

   responseStream.Close();
  }
 }
}

Sunday, April 07, 2019

Updated postcode boundaries

The boundaries for postcode areas, districts and sectors were rather out of date on my website. I originally downloaded them from a website that no longer exists. After searching around the web for updated files, I came up empty handed. The ones I found were either just as out of date as mine or had restrictive licensing attached to them.

So I set about building my own, which are now live on the site.

You can stop reading now unless you have an interest in the technical details of how I built the boundary files.

Since the freely available postcode data doesn’t include postcode unit boundaries, just the centre point of each postcode, the first step was to generate approximate boundaries for postcode units. This can be achieved using a Voronoi diagram, which generates polygons around points so that every location in the polygon is closer to the point than any other points. Although the polygons for each postcode unit are not particularly useful or accurate, combining them together at the district and sector level should give a decent approximation of the actual area covered.

The GIS tool QGIS can calculate Voronoi diagrams so I imported the postcode data into SQL Server and ran the Voronoi tool in QGIS. After several hours it hadn’t produced anything and didn’t give me any indication of when it would complete (it was stuck at 99% complete for most of that time).

So I decided to search for some code that could produce the Voronoi diagram so I could run that and would at least have an idea of where it was in the process. I found this bit of C# code that mostly worked. The output from it was edges rather than polygons, so some further work was required to construct polygons from the edges. Once the polygons were created I put those into SQL Server (adding District and Sector fields for later use) so I could work on them further in QGIS.

The next issue was that the polygons created extended far beyond the shores of the UK. This required the use of the ‘Clip vector by mask layer’ tool in QGIS, using a UK border polygon layer as the mask (note to self for next time I do this – this takes hours to complete). Then the postcode units needed to be merged into districts and sectors. The ‘Dissolve’ tool in QGIS can be used for this, using the District and Sector fields as the dissolve field.

I then had two KML files, one with all postcode districts and one with all postcode sectors. I then wrote some C# code to split the files apart so the individual districts and sectors can be displayed and downloaded.

Thursday, March 28, 2019

Sunday, February 24, 2019

UK postcode data for February 2019

I have uploaded the latest UK postcode data from the ONS to my website. I’ve run my usual sanity checks and everything looks OK but let me know if anything appears amiss. Enjoy all the 2,611,698 postcodes (alive and dead)

Thursday, January 31, 2019

Land Registry data for Dec 2018

I’ve uploaded the latest house price data from the Land Registry to my website. Currently every month feels like Groundhog Day, prices continue to rise at about 2%, the number of transactions is stable.

Thursday, January 03, 2019

Land Registry house price data November 2018

The good people at the Land Registry were probably taking a well earned vacation, so the data arrived a little later than normal but house price data for England and Wales for November 2018 is now on my site. Not much to report, annual inflation is still sitting at about 2%, the number of transactions are holding steady. Maybe things will get interesting in March, I hear something might be happening around then.

Tuesday, December 11, 2018

Free for all

As mentioned in a previous post, Google have started charging me for some more Google Maps related things but have also given me some free credit to tide me over. In fact, they think my future usage of Google Maps is going to be so high, they have given me an absolutely huge amount of credits. Since I can’t actually do anything with those credits other than spend them on the Google Maps API, I have removed the requirement for a Google Maps API key on the pages that previously required one. This will remain in place until either my credits run out or we’re close to the end of January. So please go crazy. Happy Christmas.

Important note – if the API key field contains your API key, your API key will still be used!

Saturday, November 03, 2018

More Google Maps annoyances

The other day I got another email from Google

Hi,

In June 2016, we announced a change to Maps JavaScript API requests. At that time, we gave you temporary free usage based on your consumption to ensure that your applications would continue to function. The services included in this transition period were: Elevation, Directions, Distance Matrix, Geocoding and Places.

We appreciate you as a loyal and long-standing customer. Our goal is to make sure everyone is on a simple, consistent, and scalable plan with Google Maps Platform.Starting on November 29, 2018, we will bill all your usage for Elevation, Directions, Distance Matrix, Geocoding and Places, according to our new pricing plan.

To help you with this transition, we will provide you with two months of credits, which we will automatically apply to your billing account. Please read our FAQs to understand what these credits cover and how to estimate your monthly bill.

Thank you for using Google Maps Platform.

The Google of 2016 was clearly a different company to the Google of 2018, since grandfathering of old customers when a radical change to pricing is introduced is exactly the right thing to do unlike the recent shenanigans. And generally grandfathering is a permanent thing…

So once again I’m going to have to spend some time switching things off, moving other stuff over to one of Google’s many competitors who have more sane pricing and making other things require a Google Maps API key (the route elevation page now does, sorry)

Saturday, October 27, 2018

House price data for England and Wales September 2018

I’ve uploaded the latest Land Registry data to my website. Prices continue to creep up at around 2% a year, although several regions have seen falling prices

Implementing my own version of the Google Maps Timezone API

I noticed the other day that my usage of the Google Maps Timezone API was failing. I realised this was down to me not passing in an API key with the call. In their attempts to monetise their Maps API, Google now requires the API key and each call is chargeable. So I added the key and it still didn’t work, although with a different error message. Apparently using a key with a HTTP referrer restriction wasn’t allowed.

So I decided to add a server-side handler on my server that called out to the Timezone API using a server key instead. This was fairly straightforward since it just bounced the AJAX request from the browser to the Timezone API URL.

I checked back the next day to see what my API usage looked like. I’d spent $5 in 24 hours. Continuing with that meant with my other Maps API usage I’d hit the $200 per month free limit and would have to start paying Google money again, something I’ve been loath to do since their ridiculous price increases*

I realised at that point that the Timezone API wasn’t actually doing a huge amount behind the scenes. I guessed there would be libraries out there that could do the same thing but without paying for the privilege. Turns out there is. GeoTimeZone will give the time zone ID for a location and TimeZoneConverter will convert that to a Windows TimeZoneInfo that gives me everything else I needed to build my own version of the Timezone API. The code to do that is something like this

     var lat = double.Parse(HttpContext.Current.Request.QueryString["lat"]);
     var lng = double.Parse(HttpContext.Current.Request.QueryString["lng"]);
     var tz = TimeZoneLookup.GetTimeZone(lat, lng).Result;

// get other info
var tzi = TZConvert.GetTimeZoneInfo(tz);

// write out as JSON
     var jsonObj = new JObject();
     var rawOffset = tzi.BaseUtcOffset.TotalSeconds;
     jsonObj["dstOffset"] = tzi.GetUtcOffset(DateTime.UtcNow).TotalSeconds - rawOffset;
     jsonObj["rawOffset"] = rawOffset;
     jsonObj["timeZoneId"] = tz;
     jsonObj["timeZoneName"] = tzi.StandardName;
     jsonObj["status"] = "OK";

    var json = JsonConvert.SerializeObject(jsonObj);
     HttpContext.Current.Response.Write(json);

The only thing to consider is that time zones change so it’s worth keeping the two packages up to date.

* For the record, I used to pay about $200 a month to Google. Now I pay about the same to here maps and nothing to Google. I’m intrigued to know how their new pricing has worked out for them, I’m assuming most websites would have made the same decision I did and moved somewhere else.

Friday, October 19, 2018

Downloads on Chrome

If you’re having problems downloading files from my website, it may be down to the latest release of Chrome. It seems it’s got a bug that stops downloads working in some cases. I’ve found a workaround for some downloads but there are a few places where I haven’t figured out what to do you yet. Right clicking and selecting ‘Save link as..’ may fix the issue but if that’s not offered as an option, all I can suggest is switching to a different browser

Sunday, October 07, 2018

New features on doogal.co.uk

I’ve been making a few enhancements to the website recently so thought I’d point out a few things that might be of interest.

Editable labels on geocoded locationsThe labels displayed on the map after geocoding can now be edited. Editing the label will update the text output and the KML output.

Coloured elevation charts – I’ve updated my elevation charts to include colours based on the gradient at that point of the route/segment. Here’s Box Hill in all its technicolour glory. Due to the variable quality of elevation data, it can sometimes be difficult to calculate sensible gradients, so let me know if you see anything that doesn’t look sensible.

View your starred segmentsThe segment viewer page had fairly limited functionality before so I’ve now added the ability to view all your starred segments. If you’ve connected my website to Strava on the segment explorer page, you should see a ‘View starred segments’ button which will display them all on a map. This will be quite slow to load initially if the segments are not already available on the site, but should be quicker on subsequent loads. It’s also currently limited to 100 segments.

Friday, September 28, 2018

House price data for August 2018

Another month has flown by and it’s time to upload some more Land Registry data to my website. Annual house price inflation sits at 2%, another drop that may be becoming a trend. Number of sales also appears to be on a downward trend

Thursday, August 30, 2018

House price data for July 2018

I’ve uploaded the latest house price data from the Land Registry to my website. Not much excitement in the overall figures but there’s more interesting data at the regional level. A year after being the City of Culture, Hull is suffering the hangover and the wild swings continue in central London (although this is mostly down to the small number of sales that occur in the WC postcode area).

Thursday, August 23, 2018

Decoding paths in here maps

In my panic to move my website off Google Maps, I took some shortcuts. One of those was to continue to use the helper function google.maps.geometry.encoding.decodePath. I primarily use this for decoding the paths of Strava segments, since these are encoded using the same algorithm as Google. But loading up the Google Maps API script for a single function is rather ridiculous, so I’ve converted the path decoder from the rather marvellous (and seemingly abandoned) Strava.NET library.

For your pleasure, here is some TypeScript that will create a here maps LineString from an encoded path. Other map providers are available and adjusting it for them should be fairly straightforward.

function decodePath(polylinechars: string): H.geo.LineString {
  var poly = new H.geo.LineString();

  if (polylinechars == null || polylinechars === "") {
    return poly;
  }

  var index = 0;

  var currentLat = 0;
  var currentLng = 0;

  while (index < polylinechars.length) {
    // calculate next latitude
    var sum = 0;
    var shifter = 0;
    var next5bits;
    do {
      next5bits = polylinechars.charCodeAt(index++) - 63;
      sum |= (next5bits & 31) << shifter;
      shifter += 5;
    } while (next5bits >= 32 && index < polylinechars.length);

    if (index >= polylinechars.length)
      break;

    currentLat += (sum & 1) == 1 ? ~(sum >> 1) : (sum >> 1);

    // calculate next longitude
    sum = 0;
    shifter = 0;
    do {
      next5bits = polylinechars.charCodeAt(index++) - 63;
      sum |= (next5bits & 31) << shifter;
      shifter += 5;
    } while (next5bits >= 32 && index < polylinechars.length);

    if (index >= polylinechars.length && next5bits >= 32)
      break;

    currentLng += (sum & 1) == 1 ? ~(sum >> 1) : (sum >> 1);

    poly.pushPoint(new H.geo.Point(currentLat / 100000.0, currentLng / 100000.0));
  }

  return poly;
}

Tuesday, August 21, 2018

Thursday, August 02, 2018

Here Maps full screen control

One thing missing from Here Maps is a full screen control, but fortunately it’s possible to add your own custom controls to the map. This isn’t brilliantly documented but a support person from Here Maps was able to help me out with an example. That combined with the fact I’d built something similar in Google Maps before it had its own full screen control. So here’s a TypeScript function that will add a full screen button to a Here Map.

function hereMapsFullScreenControl(ui: H.ui.UI) {
  // add custom UI control
  const myCustomControl = new H.ui.Control();
  myCustomControl.addClass("customControl");
  ui.addControl("myCustomControl", myCustomControl);
  // Set the position of the control in the UI's layout
  myCustomControl.setAlignment(H.ui.LayoutAlignment.TOP_RIGHT);

  const myCustomPanel = new H.ui.base.OverlayPanel();
  myCustomPanel.addClass("customPanel");
  myCustomControl.addChild(myCustomPanel);

  // store original styles
  const mapDiv = ui.getMap().getElement() as HTMLElement;
  let divStyle: CSSStyleDeclaration = mapDiv.style;
  if (mapDiv.runtimeStyle) {
    divStyle = mapDiv.runtimeStyle;
  }
  const originalPos: string = divStyle.position;
  let originalWidth: string = divStyle.width;
  let originalHeight: string = divStyle.height;
  // ie8 hack
  if (originalWidth === "") {
    originalWidth = mapDiv.style.width;
  }
  if (originalHeight === "") {
    originalHeight = mapDiv.style.height;
  }
  const originalTop: string = divStyle.top;
  const originalLeft: string = divStyle.left;
  const originalZIndex: string = divStyle.zIndex;
  let bodyStyle: CSSStyleDeclaration = document.body.style;
  if (document.body.runtimeStyle) {
    bodyStyle = document.body.runtimeStyle;
  }
  const originalOverflow: string = bodyStyle.overflow;

  const myCustomButton = new H.ui.base.PushButton({
    label: "Full screen",
    onStateChange: (evt) => {
      // OK, button state changed... if it's currently down
      if (myCustomButton.getState() === H.ui.base.Button.State.DOWN) {
        // go full screen
        mapDiv.style.position = "fixed";
        mapDiv.style.width = "100%";
        mapDiv.style.height = "100%";
        mapDiv.style.top = "0";
        mapDiv.style.left = "0";
        mapDiv.style.zIndex = "100";
        document.body.style.overflow = "hidden";
      } else {
        // exit full screen
        if (originalPos === "") {
          mapDiv.style.position = "relative";
        } else {
          mapDiv.style.position = originalPos;
        }
        mapDiv.style.width = originalWidth;
        mapDiv.style.height = originalHeight;
        mapDiv.style.top = originalTop;
        mapDiv.style.left = originalLeft;
        mapDiv.style.zIndex = originalZIndex;
        document.body.style.overflow = originalOverflow;
      }
      ui.getMap().getViewPort().resize();
    },
  });

  myCustomPanel.addChild(myCustomButton as any);
  myCustomPanel.setState(H.ui.base.OverlayPanel.State.OPEN);
}

Wednesday, August 01, 2018

Land Registry data for June 2018

The server is still churning through the data for the England and Wales house price data for June 2018 but the top-level data is there. Annual inflation has dropped a little, but not enough to warrant any excitement.

Saturday, July 21, 2018

The economics of the new Google Maps API pricing

The insane new pricing for the Google Maps API with a $200 per month credit leads to an odd situation. For anyone using less than $200 a month, Google Maps is probably the best option (although other map providers also generally have some kind of free tier). For anyone going over that limit, you’d have to be insane to use Google Maps since it now costs so much more than other map providers.

So if you’re in a similar situation to me and you have a bunch of pages using Google Maps, the sensible option is to convert them until you get to a stage where your Google costs are approximately $6.67 per day. The Billings Report in the Google APIs console gives a fairly up to date picture of what’s going on (turn off the ‘Include credit in costs’ option). My current costs are about $20 per day so some work still to be done.

I’d much prefer to be improving the site, but since Google have decided to be dicks, I have to rewrite stuff until I’m no longer paying them a penny…

Thursday, June 28, 2018

Tuesday, June 26, 2018

A reply to Google Maps

I received an email from Google Maps with a subject line of ‘Action needed: Contact Google Maps Platform for volume pricing’, as follows. I thought I’d translate it and reply since it came from a no-reply email address.

Hi,

We are following up on our most recent announcement. This is a reminder that Google Maps Platform’s new terms and pricing will go into effect July 16.

Translation – we are massively increasing our prices on July 16

The 2 months of credit we extended to you will automatically apply to your billing account on that date.

Translation – this won’t cover the increase in prices

You are eligible for a significantly discounted price on your monthly bill, based on your usage over the last three months.

Translation – the discounted price is not that discounted and will still be more expensive than our competitors

If you have not yet contacted us, we strongly recommend that you contact us to learn more about our volume pricing and how it can benefit you.

Reply – I did contact you. Your support staff seem kind of stressed. They sent me off to one of your partners who was only able to offer me a deal that would mean all of my advertising revenue went to Google Maps

Thank you for using Google Maps Platform.

It was fun while it lasted but I’m busily moving all my maps to another provider with more sane pricing. I would really love to know the rationale behind this move because it makes zero sense to me.

Sunday, June 10, 2018

Reducing Google Maps costs

I’m sure I’m not the only person who will be hit hard by the huge increase in the cost of Google Maps. I’ve been using Google Maps for years on my site and it’s been absolutely brilliant. But my monthly costs will increase from a couple of hundred pounds to several thousand pounds, which will wipe out any money generated through advertising. And since I was given very little notice of the increase in price, I’ve had to move quickly to try to keep my site useful but still economical. So here’s some suggestions if you’re in the same boat.

Use embedded maps – these are still free and if you’re showing a simple map with a marker then they might meet your needs. You lose Street View but as a quick temporary fix, this seems the most straightforward option.

Don’t load things immediately – I’ve been in the habit of loading maps and showing information from other Google Maps APIs as soon as the page loads, because it hardly cost anything. I’m now looking very closely at what is key information and what may only be useful to some users. If it’s only going to be useful to some users, I’m adding a button to display that data.

Turn things off – sometimes I’ve added stuff just because it looked like fun without considering whether it would be useful at all. Just removing it is generally easier than the previous option.

Other APIs – At the time of writing, my Google APIs console doesn’t even tell me the usage of some APIs that will soon cost a lot of money to use (Places and autocomplete for example). the only suggestion here is to keep an eye on that console because hopefully usage data will appear some time before we start getting charged

Move to a different map provider – If the prices of other map providers remains the same, then I’m not sure why anyone would choose Google Maps anymore. Their prices are completely out of whack with the rest of the world. But maybe they are just the first company to decide they need to charge more because actually it costs a lot to provide a mapping API. If that is the case, the other map providers will probably breathe a sigh of relief and up their prices to a similar level. And if that is the case, moving to another company could be a time-consuming and ultimately fruitless endeavour. That said, the way Google has handled this price rise has made me lose confidence in them so I’m looking at alternatives. I’ve converted one map to Here Maps fairly easily (I chose them mainly because they support KML) and will convert more as time permits

Wednesday, June 06, 2018

Goodbye Google Maps – important notice

A while back I received an email from Google Maps telling me they were introducing new pricing, but the email included the line “Based on your project usage over the last 3 months and our new pricing plan, we estimate that your new cost will be less than $200 a month and will be covered by our $200 monthly free credit” so I was pretty chuffed since I currently spend about £100 a month on Google Maps. I then promptly forgot about it.

But I then received another email saying the changes were being postponed and I thought it was time to look at how much I’d be paying. After running a few numbers, I realised I’ll have to pay several thousand pounds a month to continue to use Google Maps. Since this will wipe out any advertising revenue I make (via Google of course…), it looks like I’ll have to start removing and replacing various parts of the site.

The Google API Console is fairly useless. Although it tells me my API usage, it doesn’t tell me which pages are causing the usage so I don’t have a clear insight into what I need to remove. But two pages that are likely to be affected are the batch geocoding and batch reverse geocoding pages since they certainly generate a lot of requests. Other pages likely to be affected are random addresses, route elevation and driving distance. Other pages may fail to load maps or fail in other ways after July 16th. Apologies in advance

Wednesday, May 30, 2018

Land Registry house price data April 2018

I’ve uploaded the latest Land Registry house price data to my website. I’d read stories of annual house price falls but the Land Registry data is showing a 3% annual increase. The Land Registry numbers are a few months behind other house price indices so a fall may be on its way in a few months

Monday, May 28, 2018

One month later and TSB online banking is still a disaster

One month ago TSB tried to migrate their customers to a new online banking platform. Things didn’t go particularly well and although the media have lost interest in the problems, as a TSB customer and IT geek I’m still interested in what’s happening and thought I’d touch on a few of the issues.

The most glaring issue is that my mortgage account still doesn’t appear in my list of accounts. I assume I still have a mortgage since the payments are still being taken from my current account but there has been no sighting of it since the ‘upgrade’.

A few days ago, I tried to login, only to see an error message, <div>Sorry, we are currently experiencing technical issues. Please try again later.</div>. Not being able to login isn’t great but not totally unexpected given the recent troubles. Displaying HTML tags in your error messages is a fairly minor issue but suggests a lack of attention to detail that can lead to more serious issues down the road.

Now lets go under the hood by firing up Dev Tools in Chrome. The first error I saw was complaining about Symantec certificates which Chrome will stop supporting in a future release. Let’s hope someone at TSB renews that fairly soon before things break properly.

But the next set of errors are slightly more concerning, WebSocket connection to 'wss://127.0.0.1:3389/' failed, something on the web page is trying to connect to a server on my local machine! I can only assume this is some kind of debugging code but should definitely not be getting pushed out to their production system.

Next we have a failure to load a locale file for US English, which is odd since I’m set up for UK English. Again, nothing serious but shows a lack of attention to detail.

Finally, let’s run the page through Lighthouse, the Chrome web page checker. The performance score is a mighty 18 out of 100. This seems mostly due to Javascript and CSS that isn’t bundled together and isn’t being loaded using async or defer, meaning everything has to load before the page can be displayed.

And that’s what I found spending a few minutes on the TSB website, who knows what could be found if I was an evil hacker wanting to steal some money

Sunday, May 27, 2018

Converting KML file to shapefiles

For a long time I’ve been searching for some code that I can use on my website to convert the various KML postcode files that are available. Since many of these files are generated on the fly, I didn’t want to use a one off convertor. But I’ve always failed to find anything that will do the job. Since I don’t know the internals of shapefiles, it’s not something I wanted to write myself.

So I’ve kind of given up, but if you need to convert any of the KML files yourself, then this tool may be useful. From my admittedly not very thorough testing, it seems to do a good job of the conversion

Thursday, May 03, 2018

Land Registry house price data March 2018

I’ve uploaded the latest property data for England and Wales from the Land Registry to my website. Annual house price inflation has been slowly creeping up and now stands at 3%. Sales data is harder to get a handle on since it can take months for all the sales data to come through but it looks like sale may be dropping off a little.

Wednesday, March 28, 2018

Land Registry property sales data for February 2018

My server is still crunching through some of the numbers but most of the England and Wales house price data for February 2018 is up on my site. There’s very little to report, annual inflation remains at just short of 3%, number of transactions remains at a steady level

Thursday, March 08, 2018

My response to every anti-cycling article ever

If it’s not some Daily Mail article complaining about cyclists not using a cycle lane, it’s someone starting up a petition demanding cyclists should pay Road Tax (sorry no links but I don’t want to give them any encouragement). But one thing remains the same, the arguments put forward in the comments sections of these articles are always the same tired ill-thought out complaints. So rather than repeat myself on every article, here are my responses to the most common issues

Cyclists should pay road tax

The most common response to this is that road tax doesn’t exist, which is accurate, if a little pedantic. But we all presumably understand that Vehicle Excise Duty is charged based on the pollution caused by the vehicle, and I presume we can all agree the pollution generated by a bicycle is pretty much zero and hence the VED payable by the cyclist would be £0? OK, make us pay VED, I have no problem with that.

Beyond that, like most cyclists I own a car, for which I’m paying VED. I also pay Council Tax, Income Tax and VAT. I’ve just paid a big chunk of change in Stamp Duty to buy a house. Like most cyclists, I’m already contributing whilst helping to reduce congestion by using a bike instead of my car for every journey.

All cyclists jump red lights

Yes, some cyclists jump red lights, just like some drivers jump red lights. Just like some pedestrians do stupid things like crossing the road without looking. Cyclists are not an homogenous group. I suspect the problem here is that drivers feel that cyclists get away with jumping red lights. Here’s the thing, if a cyclist jumps a red light and causes an accident, the chances are he’ll come off worse or if he hits a pedestrian is unlikely to cause life changing injury. If someone in 2 tonnes of steel causes an accident, the chances of injury or death are much higher. So, with limited resources, the police are going to concentrate on the potentially more dangerous group of road users.

Cyclists should use the cycle lane when one is available

First up, there is no legal requirement for a cyclist to use a cycle lane if one is available. You can argue about whether that should be the case, but it is. But more importantly, a lot of cycle lanes are not fit for purpose. Take this example of road rage on Priory Lane in South West London. The driver gets insanely upset that the cyclist isn’t using the cycle lane (how stressful is driving BTW, get a bike, you’ll have much more fun…) but the cycle lane has a couple of problems. You have to give way at every junction along the road, of which there are quite a few. And when travelling South, it’s on the wrong side of the road. So if you are a confident cyclist, chances are you’ll use the road.

Or how about this 5 metre bit of cycle lane. Am I really expected to use that?

But these cycle lanes probably do work for some less confident cyclists and that is perhaps what drivers need to understand. Cycle lanes are not built for every cyclist, some are there to encourage more people to cycle, because that would be a good thing, right?

Cyclists should be required to have insurance

I’m almost in agreement with this. I’d have no problem getting insurance. But then what about the kid who has just got a shiny new bike for Christmas, they aren’t allowed to use it because their dad forgot to get insurance? Again, I am assuming encouraging cycling is the aim here

Cyclists should have number plates

Yeh, why not, if that floats your boat…

Saturday, March 03, 2018

Converting Strava segments to FIT files

Strava Premium users who own a Garmin can sync their favourite Strava segments with their Garmin device automatically. But that’s not an option available to free users. Apparently this site used to be able to generate the FIT files used by Garmins but Strava made some fairly radical changes to their API that broke that site (and many others).

So I’ve added a FIT file export to my Segment Explorer. Due to rather inclement weather I’ve not been able to test this out on the road yet, but downloading the FIT file and shoving it in my Garmin\NewFiles folder displays it in my Segments list so it seems to work. If you authorize my app then your PR time will also be included in the file.

Update – You can now add a goal to the generated FIT file if you generate the file from the segment details page

Wednesday, February 28, 2018

Land Registry data for January 2018

I’ve uploaded the latest property data for England and Wales to my website. Like a broken record, I can report that house prices continue their gradual rise ever upwards. Sales volumes are holding steady.

Wednesday, February 07, 2018

Wednesday, January 03, 2018

Sunday, November 26, 2017

Sunday, October 08, 2017

Adjusting Strava KOM speeds for gradient

I’ve been wanting to improve the algorithm behind my Strava segment chooser for some time. Until now it’s been based off the number of riders who’ve attempted the segment and the power output of the KOM. It gave reasonable results but had to ignore any segments whose KOM didn’t have any power data recorded (either from a power meter or estimated based off the rider’s weight and speed). And although more power means more speed, a heavier rider will have to output more power to go at the same speed so the power output isn’t the perfect indicator of how difficult a KOM is to beat.

But using speed also has its problems. 20 mph on a flat segment is generally not going to win any KOMs but 20mph on a 10% gradient may well do. Adjusting the KOM speed for the gradient of the segment has been my aim for a while, but I haven’t had a chance to look at it until now.

Since I now have quite a few segments in my database, I thought it was time to analyse KOM speeds against gradient. I put each segment in a bucket based on its gradient and calculated the average KOM speed. This gave me the following, rather surprising graph.

image

For gradients between about –2% and 10%, things work as I’d expect. The steeper the gradient, the slower the KOMs are. Beyond that, things get a bit weird. Steeper uphill segments get a little faster and steeper downhill segments get a little slower. I have a few ideas about why this might be (more mountain bike segments, the terror induced by a steep downhill) but nothing definitive.

Whatever the reason, I decided to ignore segments outside –2% to 10%. The graph in this range can be approximated to a straight line so calculating an adjusted speed is fairly easy. I’ve applied this to my segment chooser and the results are, er, much the same as before…

Friday, September 22, 2017

Convert a list to a comma string

It’s nothing special, but I’ve found this useful in the past and probably will again so here for my memory bank is a function that converts a list to a comma string


using System.Collections.Generic;
using System.Text;

namespace Utils
{
  public static class ListHelper
  {
    public static string ToCommaString<T>(this IEnumerable<T> list, string separator = ", ")
    {
      var builder = new StringBuilder();

      foreach (var t in list)
      {
        if (builder.Length > 0)
          builder.Append(separator);
        builder.Append(t);
      }

      return builder.ToString();
    }
  }
}

Friday, July 28, 2017

Wednesday, June 28, 2017

Land Registry data for May 2017

I’ve uploaded the latest Land Registry data for England and Wales to my website. My earlier predictions of house price growth going negative appear to be incorrect. Prices continue to go up but at an ever decreasing pace.

Monday, June 26, 2017

Recovering data from a crashed Garmin 810

I love my Garmin and it would be a perfect piece of tech if it weren’t for one problem. Almost every time I use it to follow a course, it will die at some point during the ride. I can turn it back on and after several minutes of confusion it will figure out where it is and I can resume recording. The suggestions on the Garmin forum mostly seem to be to downgrade to an older version of the firmware, but I’m too lazy to do that and keep hoping Garmin will fix it at some point.

But they haven’t yet, so I just live with it. So I thought I’d document what I do to recover the data from before the crash. When I plug the Garmin in, the data from after the crash will sync to Strava but it will also complain that something failed to sync. So the first thing to do is to manually upload the pre-crash data to Strava. This will be a .fit file in the Garmin\Activities folder of the device. Strava seems to be able to cope with the file, even though the Garmin sync can’t.

You may be happy enough just doing that, but I like to have an accurate record of my rides so I want to merge the two together. So I now

  • Export the two activities from Strava as GPX files.
  • Open them both in this tool.
  • Select both files, right click and merge them
  • In the newly created file, select both tracks, right click and select Restructuring/Join
  • Save the new file
  • Delete the two original activities from Strava
  • Upload the newly created file to Strava and feel satisfied with all the PRs I’ve hopefully achieved…

Wednesday, May 31, 2017

Property data for England and Wales April 2017

I’ve uploaded the April 2017 Land Registry data to my website. The annual change is still positive and although it’s heading towards zero, the progress is slowing.

Tuesday, May 30, 2017

Thursday, May 04, 2017

Land Registry house price data for March 2017

Another month passes by and we have a new batch of house price data from the Land Registry. The annual change remains positive but continues to creep downwards. In some areas, the annual change has gone negative, you can see more detailed figures here.

Tuesday, March 28, 2017

Land Registry house price data February 2017

The latest Land Registry sales data for England and Wales has been released and my server is currently churning through the data. The top level annual change remains positive but continues to trend downwards.

Thursday, March 02, 2017

Postcode data for February 2017

The ONS has released its latest postcode data, which is now on my site. I’ve run my usual sanity checks and it all seems to be in order, but let me know if you spot any anomalies.

Tuesday, February 28, 2017

Land Registry data for January 2017

Another month has flown by and it’s time for a house price update from the Land Registry. Sale volumes remain stable and prices continue to rise at an ever slower rate. It looks like the annual change will turn negative in 5 months if things continue in the same way.

Tuesday, January 31, 2017

Friday, December 30, 2016

Property sale data November 2016

I’ve uploaded the latest house price data from the Land Registry for England and Wales to my website. I’ve also made some changes to the way annual changes are calculated. This means the annual change figure is much less volatile, but it does mean changes in the directions of travel are slower to appear. The current annual change is 3.5% but reducing each month. If the current trend continues, the annual change will go negative in about 9 months time.

UK stations updated with 2016 usage data

I’ve uploaded the UK station usage data for 2016 to my site. I’ve also made a few minor tweaks to the layout which hopefully make the data more useful

Wednesday, November 30, 2016

Monday, November 28, 2016

Land Registry house price data for October 2016

I’ve uploaded the October 2016 Land Registry house price data to my website. Prices are just about managing to maintain a positive annual increase. Although houses are continuing their upward march, there seems to have been a mini collapse in the price of flats, I’m guessing due to the Buy To Let tax changes.

Friday, October 28, 2016

Land Registry data for September 2016

I’ve uploaded the latest house price data to my website. Prices keep creeping upwards, although there are lots of regional variations.

Friday, September 30, 2016

England and Wales house price data August 2016

As I type this, my server is churning through the latest Land Registry data for August 2016. The top level data has been imported and it suggests prices continue to glide ever upwards. The only difference seems to be the annual change is getting smaller, whereas it’s been around 5% for the last few years, it’s now around 2%.

If you look back at the data for earlier this year, you can now clearly see the spike in sales in March just before the buy to let Stamp Duty hike. Sales now seem to have returned to the levels seen after the financial crisis.

Wednesday, August 31, 2016

Tuesday, August 23, 2016

UK Postcode data August 2016

I’ve uploaded the latest data from the ONS to my website. I have run my usual sanity checks, let me know if you spot anything amiss

Wednesday, August 17, 2016

Google Maps Distance Matrix may not be what you’re after

For many years I’ve been using the Google Maps APIs on my website. It’s been fun to use and until recently the licensing has been very unrestrictive. If an API returned a response saying you’d gone over the query limit, just wait for a second or so, try again and generally it would work. So with some use of setTimeout, it was possible to build reasonably scalable apps that cost nothing.

It’s looking like those days are coming to an end. The various APIs are starting to introduce hard limits on their usage. Once you’re over the limit, that’s it until the counter resets at the start of the next day. I first hit this with my use of the Directions API in my Driving Distances page. I can’t say I’m too happy with the way it was introduced, the Google API Console had given no previous indication of my usage of the API but on the same day as they started displaying the usage report, the hard limit was also introduced. Since my site was way over the limit, that page fell over almost immediately.

So I had a Baldrick cunning plan. I’d swap out the Directions API for the Distance Matrix API. This is exactly the kind of application this API was designed for. Unfortunately I failed to read the usage limits correctly and after I uploaded the new code, the page fell over in a heap again after a few hours. It turns out the usage limits apply to the elements passed to the Distance Matrix API, not the number of requests. So a 10 by 10 matrix counts as 100 towards the free 2,500 limit, not 1 as I had assumed. Given that this API provides less information and has fewer options than the Directions API but has the exact same usage limits, this is rather disappointing!

I am trying to figure out the best way forward now. I could start to pay for extra requests, but since a 100 by 100 matrix would cost $5, the costs could mount up quickly. I can put a maximum daily cost on the account so I don’t have to pay enormous amounts if someone overuses the page, but this could lead to the page becoming unavailable again.

I suspect the outcome will be me removing the page, or at least no longer linking to it from the rest of the site. I make a tidy sum from Google AdSense advertising on the site, but I fear this may just be the start, as more and more APIs start to introduce a hard limit and I don’t particularly want to pay that money back to Google every month to pay for their APIs. It was fun whilst it lasted I guess!

Saturday, July 09, 2016

textareas slow in Chrome

So there’s a page on my site, https://www.doogal.co.uk/BatchGeocoding.php, that someone complained about. Specifically, they complained that if they tried to geocode 3,000 postcodes, it was terribly slow. I tried it myself and experienced the same problem. When geocoding postcodes, the page uses my own internal database, so it should suffer none of the throttling issues of Google Maps. No worries I thought, I can reproduce the problem, which is generally the biggest hurdle, fixing it should be straightforward.

So I fired up Chrome’s profiler and found… absolutely nothing… None of the delays were in my code. So I tried Microsoft Edge and it was super quick. I pretty much gave up at that point and suggested the user tried MS Edge.

Three weeks later I had a look with fresh eyes. And something popped up from the recesses of my mind, spellcheck=”false”. I vaguely remembered setting that attribute on a textarea in the past had improved performance and once again, this fixed the issue. A single geocode was previously taking a second, now all 3,000 took a couple of minutes. This may be a bug in Chrome or maybe spellchecking is a very CPU intensive process. Either way, turning it off makes everything better.

As always this is just a reminder for me and maybe it will be useful to someone passing through.

Friday, July 01, 2016