Free for all

As mentioned in a previous post, Google have started charging me for some more Google Maps related things but have also given me some free credit to tide me over. In fact, they think my future usage of Google Maps is going to be so high, they have given me an absolutely huge amount of credits. Since I can’t actually do anything with those credits other than spend them on the Google Maps API, I have removed the requirement for a Google Maps API key on the pages that previously required one. This will remain in place until either my credits run out or we’re close to the end of January. So please go crazy. Happy Christmas.

Important note – if the API key field contains your API key, your API key will still be used!

Land Registry house price data for October 2018

The latest house price data from the Land Registry is now on my site. House prices continue to rise slowly, although trending lower currently.

UK postcode data November 2018

I’ve updated my website with the latest ONS postcode data for November 2018. As ever if you spot anything amiss, let me know

More Google Maps annoyances

The other day I got another email from Google

Hi,

In June 2016, we announced a change to Maps JavaScript API requests. At that time, we gave you temporary free usage based on your consumption to ensure that your applications would continue to function. The services included in this transition period were: Elevation, Directions, Distance Matrix, Geocoding and Places.

We appreciate you as a loyal and long-standing customer. Our goal is to make sure everyone is on a simple, consistent, and scalable plan with Google Maps Platform.Starting on November 29, 2018, we will bill all your usage for Elevation, Directions, Distance Matrix, Geocoding and Places, according to our new pricing plan.

To help you with this transition, we will provide you with two months of credits, which we will automatically apply to your billing account. Please read our FAQs to understand what these credits cover and how to estimate your monthly bill.

Thank you for using Google Maps Platform.

The Google of 2016 was clearly a different company to the Google of 2018, since grandfathering of old customers when a radical change to pricing is introduced is exactly the right thing to do unlike the recent shenanigans. And generally grandfathering is a permanent thing…

So once again I’m going to have to spend some time switching things off, moving other stuff over to one of Google’s many competitors who have more sane pricing and making other things require a Google Maps API key (the route elevation page now does, sorry)

House price data for England and Wales September 2018

I’ve uploaded the latest Land Registry data to my website. Prices continue to creep up at around 2% a year, although several regions have seen falling prices

Implementing my own version of the Google Maps Timezone API

I noticed the other day that my usage of the Google Maps Timezone API was failing. I realised this was down to me not passing in an API key with the call. In their attempts to monetise their Maps API, Google now requires the API key and each call is chargeable. So I added the key and it still didn’t work, although with a different error message. Apparently using a key with a HTTP referrer restriction wasn’t allowed.

So I decided to add a server-side handler on my server that called out to the Timezone API using a server key instead. This was fairly straightforward since it just bounced the AJAX request from the browser to the Timezone API URL.

I checked back the next day to see what my API usage looked like. I’d spent $5 in 24 hours. Continuing with that meant with my other Maps API usage I’d hit the $200 per month free limit and would have to start paying Google money again, something I’ve been loath to do since their ridiculous price increases*

I realised at that point that the Timezone API wasn’t actually doing a huge amount behind the scenes. I guessed there would be libraries out there that could do the same thing but without paying for the privilege. Turns out there is. GeoTimeZone will give the time zone ID for a location and TimeZoneConverter will convert that to a Windows TimeZoneInfo that gives me everything else I needed to build my own version of the Timezone API. The code to do that is something like this

     var lat = double.Parse(HttpContext.Current.Request.QueryString["lat"]);
     var lng = double.Parse(HttpContext.Current.Request.QueryString["lng"]);
     var tz = TimeZoneLookup.GetTimeZone(lat, lng).Result;
// get other info
var tzi = TZConvert.GetTimeZoneInfo(tz);
     // write out as JSON
     var jsonObj = new JObject();
     var rawOffset = tzi.BaseUtcOffset.TotalSeconds;
     jsonObj["dstOffset"] = tzi.GetUtcOffset(DateTime.UtcNow).TotalSeconds - rawOffset;
     jsonObj["rawOffset"] = rawOffset;
     jsonObj["timeZoneId"] = tz;
     jsonObj["timeZoneName"] = tzi.StandardName;
     jsonObj["status"] = "OK";
    var json = JsonConvert.SerializeObject(jsonObj);
     HttpContext.Current.Response.Write(json);

The only thing to consider is that time zones change so it’s worth keeping the two packages up to date.

* For the record, I used to pay about $200 a month to Google. Now I pay about the same to here maps and nothing to Google. I’m intrigued to know how their new pricing has worked out for them, I’m assuming most websites would have made the same decision I did and moved somewhere else.

Downloads on Chrome

If you’re having problems downloading files from my website, it may be down to the latest release of Chrome. It seems it’s got a bug that stops downloads working in some cases. I’ve found a workaround for some downloads but there are a few places where I haven’t figured out what to do you yet. Right clicking and selecting ‘Save link as..’ may fix the issue but if that’s not offered as an option, all I can suggest is switching to a different browser

New features on doogal.co.uk

I’ve been making a few enhancements to the website recently so thought I’d point out a few things that might be of interest.

Editable labels on geocoded locations – The labels displayed on the map after geocoding can now be edited. Editing the label will update the text output and the KML output.

Coloured elevation charts – I’ve updated my elevation charts to include colours based on the gradient at that point of the route/segment. Here’s Box Hill in all its technicolour glory. Due to the variable quality of elevation data, it can sometimes be difficult to calculate sensible gradients, so let me know if you see anything that doesn’t look sensible.

View your starred segments – The segment viewer page had fairly limited functionality before so I’ve now added the ability to view all your starred segments. If you’ve connected my website to Strava on the segment explorer page, you should see a ‘View starred segments’ button which will display them all on a map. This will be quite slow to load initially if the segments are not already available on the site, but should be quicker on subsequent loads. It’s also currently limited to 100 segments.

House price data for August 2018

Another month has flown by and it’s time to upload some more Land Registry data to my website. Annual house price inflation sits at 2%, another drop that may be becoming a trend. Number of sales also appears to be on a downward trend

House price data for July 2018

I’ve uploaded the latest house price data from the Land Registry to my website. Not much excitement in the overall figures but there’s more interesting data at the regional level. A year after being the City of Culture, Hull is suffering the hangover and the wild swings continue in central London (although this is mostly down to the small number of sales that occur in the WC postcode area).

Decoding paths in here maps

In my panic to move my website off Google Maps, I took some shortcuts. One of those was to continue to use the helper function google.maps.geometry.encoding.decodePath. I primarily use this for decoding the paths of Strava segments, since these are encoded using the same algorithm as Google. But loading up the Google Maps API script for a single function is rather ridiculous, so I’ve converted the path decoder from the rather marvellous (and seemingly abandoned) Strava.NET library.

For your pleasure, here is some TypeScript that will create a here maps LineString from an encoded path. Other map providers are available and adjusting it for them should be fairly straightforward.

function decodePath(polylinechars: string): H.geo.LineString {
  var poly = new H.geo.LineString();

  if (polylinechars == null || polylinechars === "") {
    return poly;
  }

  var index = 0;

  var currentLat = 0;
  var currentLng = 0;

  while (index < polylinechars.length) {
    // calculate next latitude
    var sum = 0;
    var shifter = 0;
    var next5bits;
    do {
      next5bits = polylinechars.charCodeAt(index++) - 63;
      sum |= (next5bits & 31) << shifter;
      shifter += 5;
    } while (next5bits >= 32 && index < polylinechars.length);

    if (index >= polylinechars.length)
      break;

    currentLat += (sum & 1) == 1 ? ~(sum >> 1) : (sum >> 1);

    // calculate next longitude
    sum = 0;
    shifter = 0;
    do {
      next5bits = polylinechars.charCodeAt(index++) - 63;
      sum |= (next5bits & 31) << shifter;
      shifter += 5;
    } while (next5bits >= 32 && index < polylinechars.length);

    if (index >= polylinechars.length && next5bits >= 32)
      break;

    currentLng += (sum & 1) == 1 ? ~(sum >> 1) : (sum >> 1);

    poly.pushPoint(new H.geo.Point(currentLat / 100000.0, currentLng / 100000.0));
  }

  return poly;
}

UK postcode date August 2018

I’ve uploaded the latest postcode data from the ONS to my website. I’ve checked the data with my usual sanity check but let me know if you spot anything amiss

Here Maps full screen control

One thing missing from Here Maps is a full screen control, but fortunately it’s possible to add your own custom controls to the map. This isn’t brilliantly documented but a support person from Here Maps was able to help me out with an example. That combined with the fact I’d built something similar in Google Maps before it had its own full screen control. So here’s a TypeScript function that will add a full screen button to a Here Map.

function hereMapsFullScreenControl(ui: H.ui.UI) {
  // add custom UI control
  const myCustomControl = new H.ui.Control();
  myCustomControl.addClass("customControl");
  ui.addControl("myCustomControl", myCustomControl);
  // Set the position of the control in the UI's layout
  myCustomControl.setAlignment(H.ui.LayoutAlignment.TOP_RIGHT);

  const myCustomPanel = new H.ui.base.OverlayPanel();
  myCustomPanel.addClass("customPanel");
  myCustomControl.addChild(myCustomPanel);

  // store original styles
  const mapDiv = ui.getMap().getElement() as HTMLElement;
  let divStyle: CSSStyleDeclaration = mapDiv.style;
  if (mapDiv.runtimeStyle) {
    divStyle = mapDiv.runtimeStyle;
  }
  const originalPos: string = divStyle.position;
  let originalWidth: string = divStyle.width;
  let originalHeight: string = divStyle.height;
  // ie8 hack
  if (originalWidth === "") {
    originalWidth = mapDiv.style.width;
  }
  if (originalHeight === "") {
    originalHeight = mapDiv.style.height;
  }
  const originalTop: string = divStyle.top;
  const originalLeft: string = divStyle.left;
  const originalZIndex: string = divStyle.zIndex;
  let bodyStyle: CSSStyleDeclaration = document.body.style;
  if (document.body.runtimeStyle) {
    bodyStyle = document.body.runtimeStyle;
  }
  const originalOverflow: string = bodyStyle.overflow;

  const myCustomButton = new H.ui.base.PushButton({
    label: "Full screen",
    onStateChange: (evt) => {
      // OK, button state changed... if it's currently down
      if (myCustomButton.getState() === H.ui.base.Button.State.DOWN) {
        // go full screen
        mapDiv.style.position = "fixed";
        mapDiv.style.width = "100%";
        mapDiv.style.height = "100%";
        mapDiv.style.top = "0";
        mapDiv.style.left = "0";
        mapDiv.style.zIndex = "100";
        document.body.style.overflow = "hidden";
      } else {
        // exit full screen
        if (originalPos === "") {
          mapDiv.style.position = "relative";
        } else {
          mapDiv.style.position = originalPos;
        }
        mapDiv.style.width = originalWidth;
        mapDiv.style.height = originalHeight;
        mapDiv.style.top = originalTop;
        mapDiv.style.left = originalLeft;
        mapDiv.style.zIndex = originalZIndex;
        document.body.style.overflow = originalOverflow;
      }
      ui.getMap().getViewPort().resize();
    },
  });

  myCustomPanel.addChild(myCustomButton as any);
  myCustomPanel.setState(H.ui.base.OverlayPanel.State.OPEN);
}

Land Registry data for June 2018

The server is still churning through the data for the England and Wales house price data for June 2018 but the top-level data is there. Annual inflation has dropped a little, but not enough to warrant any excitement.

The economics of the new Google Maps API pricing

The insane new pricing for the Google Maps API with a $200 per month credit leads to an odd situation. For anyone using less than $200 a month, Google Maps is probably the best option (although other map providers also generally have some kind of free tier). For anyone going over that limit, you’d have to be insane to use Google Maps since it now costs so much more than other map providers.

So if you’re in a similar situation to me and you have a bunch of pages using Google Maps, the sensible option is to convert them until you get to a stage where your Google costs are approximately $6.67 per day. The Billings Report in the Google APIs console gives a fairly up to date picture of what’s going on (turn off the ‘Include credit in costs’ option). My current costs are about $20 per day so some work still to be done.

I’d much prefer to be improving the site, but since Google have decided to be dicks, I have to rewrite stuff until I’m no longer paying them a penny…

House price data May 2018

The Land Registry data for May 2018 is now on my website. House prices continue to fail to do anything exciting, continuing their slow ascent upwards.

A reply to Google Maps

I received an email from Google Maps with a subject line of ‘Action needed: Contact Google Maps Platform for volume pricing’, as follows. I thought I’d translate it and reply since it came from a no-reply email address.

Hi,

We are following up on our most recent announcement. This is a reminder that Google Maps Platform’s new terms and pricing will go into effect July 16.

Translation – we are massively increasing our prices on July 16

The 2 months of credit we extended to you will automatically apply to your billing account on that date.

Translation – this won’t cover the increase in prices

You are eligible for a significantly discounted price on your monthly bill, based on your usage over the last three months.

Translation – the discounted price is not that discounted and will still be more expensive than our competitors

If you have not yet contacted us, we strongly recommend that you contact us to learn more about our volume pricing and how it can benefit you.

Reply – I did contact you. Your support staff seem kind of stressed. They sent me off to one of your partners who was only able to offer me a deal that would mean all of my advertising revenue went to Google Maps

Thank you for using Google Maps Platform.

It was fun while it lasted but I’m busily moving all my maps to another provider with more sane pricing. I would really love to know the rationale behind this move because it makes zero sense to me.

Reducing Google Maps costs

I’m sure I’m not the only person who will be hit hard by the huge increase in the cost of Google Maps. I’ve been using Google Maps for years on my site and it’s been absolutely brilliant. But my monthly costs will increase from a couple of hundred pounds to several thousand pounds, which will wipe out any money generated through advertising. And since I was given very little notice of the increase in price, I’ve had to move quickly to try to keep my site useful but still economical. So here’s some suggestions if you’re in the same boat.

Use embedded maps – these are still free and if you’re showing a simple map with a marker then they might meet your needs. You lose Street View but as a quick temporary fix, this seems the most straightforward option.

Don’t load things immediately – I’ve been in the habit of loading maps and showing information from other Google Maps APIs as soon as the page loads, because it hardly cost anything. I’m now looking very closely at what is key information and what may only be useful to some users. If it’s only going to be useful to some users, I’m adding a button to display that data.

Turn things off – sometimes I’ve added stuff just because it looked like fun without considering whether it would be useful at all. Just removing it is generally easier than the previous option.

Other APIs – At the time of writing, my Google APIs console doesn’t even tell me the usage of some APIs that will soon cost a lot of money to use (Places and autocomplete for example). the only suggestion here is to keep an eye on that console because hopefully usage data will appear some time before we start getting charged

Move to a different map provider – If the prices of other map providers remains the same, then I’m not sure why anyone would choose Google Maps anymore. Their prices are completely out of whack with the rest of the world. But maybe they are just the first company to decide they need to charge more because actually it costs a lot to provide a mapping API. If that is the case, the other map providers will probably breathe a sigh of relief and up their prices to a similar level. And if that is the case, moving to another company could be a time-consuming and ultimately fruitless endeavour. That said, the way Google has handled this price rise has made me lose confidence in them so I’m looking at alternatives. I’ve converted one map to Here Maps fairly easily (I chose them mainly because they support KML) and will convert more as time permits

Goodbye Google Maps – important notice

A while back I received an email from Google Maps telling me they were introducing new pricing, but the email included the line “Based on your project usage over the last 3 months and our new pricing plan, we estimate that your new cost will be less than $200 a month and will be covered by our $200 monthly free credit” so I was pretty chuffed since I currently spend about £100 a month on Google Maps. I then promptly forgot about it.

But I then received another email saying the changes were being postponed and I thought it was time to look at how much I’d be paying. After running a few numbers, I realised I’ll have to pay several thousand pounds a month to continue to use Google Maps. Since this will wipe out any advertising revenue I make (via Google of course…), it looks like I’ll have to start removing and replacing various parts of the site.

The Google API Console is fairly useless. Although it tells me my API usage, it doesn’t tell me which pages are causing the usage so I don’t have a clear insight into what I need to remove. But two pages that are likely to be affected are the batch geocoding and batch reverse geocoding pages since they certainly generate a lot of requests. Other pages likely to be affected are random addresses, route elevation and driving distance. Other pages may fail to load maps or fail in other ways after July 16th. Apologies in advance

UK postcode data for May 2018

I’ve uploaded the latest UK postcode data to my website. I’ve done my usual sanity checks but if you spot anything amiss, let me know

Land Registry house price data April 2018

I’ve uploaded the latest Land Registry house price data to my website. I’d read stories of annual house price falls but the Land Registry data is showing a 3% annual increase. The Land Registry numbers are a few months behind other house price indices so a fall may be on its way in a few months

One month later and TSB online banking is still a disaster

One month ago TSB tried to migrate their customers to a new online banking platform. Things didn’t go particularly well and although the media have lost interest in the problems, as a TSB customer and IT geek I’m still interested in what’s happening and thought I’d touch on a few of the issues.

The most glaring issue is that my mortgage account still doesn’t appear in my list of accounts. I assume I still have a mortgage since the payments are still being taken from my current account but there has been no sighting of it since the ‘upgrade’.

A few days ago, I tried to login, only to see an error message, <div>Sorry, we are currently experiencing technical issues. Please try again later.</div>. Not being able to login isn’t great but not totally unexpected given the recent troubles. Displaying HTML tags in your error messages is a fairly minor issue but suggests a lack of attention to detail that can lead to more serious issues down the road.

Now lets go under the hood by firing up Dev Tools in Chrome. The first error I saw was complaining about Symantec certificates which Chrome will stop supporting in a future release. Let’s hope someone at TSB renews that fairly soon before things break properly.

But the next set of errors are slightly more concerning, WebSocket connection to 'wss://127.0.0.1:3389/' failed, something on the web page is trying to connect to a server on my local machine! I can only assume this is some kind of debugging code but should definitely not be getting pushed out to their production system.

Next we have a failure to load a locale file for US English, which is odd since I’m set up for UK English. Again, nothing serious but shows a lack of attention to detail.

Finally, let’s run the page through Lighthouse, the Chrome web page checker. The performance score is a mighty 18 out of 100. This seems mostly due to Javascript and CSS that isn’t bundled together and isn’t being loaded using async or defer, meaning everything has to load before the page can be displayed.

And that’s what I found spending a few minutes on the TSB website, who knows what could be found if I was an evil hacker wanting to steal some money

Converting KML file to shapefiles

For a long time I’ve been searching for some code that I can use on my website to convert the various KML postcode files that are available. Since many of these files are generated on the fly, I didn’t want to use a one off convertor. But I’ve always failed to find anything that will do the job. Since I don’t know the internals of shapefiles, it’s not something I wanted to write myself.

So I’ve kind of given up, but if you need to convert any of the KML files yourself, then this tool may be useful. From my admittedly not very thorough testing, it seems to do a good job of the conversion

Land Registry house price data March 2018

I’ve uploaded the latest property data for England and Wales from the Land Registry to my website. Annual house price inflation has been slowly creeping up and now stands at 3%. Sales data is harder to get a handle on since it can take months for all the sales data to come through but it looks like sale may be dropping off a little.

Land Registry property sales data for February 2018

My server is still crunching through some of the numbers but most of the England and Wales house price data for February 2018 is up on my site. There’s very little to report, annual inflation remains at just short of 3%, number of transactions remains at a steady level

My response to every anti-cycling article ever

If it’s not some Daily Mail article complaining about cyclists not using a cycle lane, it’s someone starting up a petition demanding cyclists should pay Road Tax (sorry no links but I don’t want to give them any encouragement). But one thing remains the same, the arguments put forward in the comments sections of these articles are always the same tired ill-thought out complaints. So rather than repeat myself on every article, here are my responses to the most common issues

Cyclists should pay road tax

The most common response to this is that road tax doesn’t exist, which is accurate, if a little pedantic. But we all presumably understand that Vehicle Excise Duty is charged based on the pollution caused by the vehicle, and I presume we can all agree the pollution generated by a bicycle is pretty much zero and hence the VED payable by the cyclist would be £0? OK, make us pay VED, I have no problem with that.

Beyond that, like most cyclists I own a car, for which I’m paying VED. I also pay Council Tax, Income Tax and VAT. I’ve just paid a big chunk of change in Stamp Duty to buy a house. Like most cyclists, I’m already contributing whilst helping to reduce congestion by using a bike instead of my car for every journey.

All cyclists jump red lights

Yes, some cyclists jump red lights, just like some drivers jump red lights. Just like some pedestrians do stupid things like crossing the road without looking. Cyclists are not an homogenous group. I suspect the problem here is that drivers feel that cyclists get away with jumping red lights. Here’s the thing, if a cyclist jumps a red light and causes an accident, the chances are he’ll come off worse or if he hits a pedestrian is unlikely to cause life changing injury. If someone in 2 tonnes of steel causes an accident, the chances of injury or death are much higher. So, with limited resources, the police are going to concentrate on the potentially more dangerous group of road users.

Cyclists should use the cycle lane when one is available

First up, there is no legal requirement for a cyclist to use a cycle lane if one is available. You can argue about whether that should be the case, but it is. But more importantly, a lot of cycle lanes are not fit for purpose. Take this example of road rage on Priory Lane in South West London. The driver gets insanely upset that the cyclist isn’t using the cycle lane (how stressful is driving BTW, get a bike, you’ll have much more fun…) but the cycle lane has a couple of problems. You have to give way at every junction along the road, of which there are quite a few. And when travelling South, it’s on the wrong side of the road. So if you are a confident cyclist, chances are you’ll use the road.

Or how about this 5 metre bit of cycle lane. Am I really expected to use that?

But these cycle lanes probably do work for some less confident cyclists and that is perhaps what drivers need to understand. Cycle lanes are not built for every cyclist, some are there to encourage more people to cycle, because that would be a good thing, right?

Cyclists should be required to have insurance

I’m almost in agreement with this. I’d have no problem getting insurance. But then what about the kid who has just got a shiny new bike for Christmas, they aren’t allowed to use it because their dad forgot to get insurance? Again, I am assuming encouraging cycling is the aim here

Cyclists should have number plates

Yeh, why not, if that floats your boat…

Converting Strava segments to FIT files

Strava Premium users who own a Garmin can sync their favourite Strava segments with their Garmin device automatically. But that’s not an option available to free users. Apparently this site used to be able to generate the FIT files used by Garmins but Strava made some fairly radical changes to their API that broke that site (and many others).

So I’ve added a FIT file export to my Segment Explorer. Due to rather inclement weather I’ve not been able to test this out on the road yet, but downloading the FIT file and shoving it in my Garmin\NewFiles folder displays it in my Segments list so it seems to work. If you authorize my app then your PR time will also be included in the file.

Update – You can now add a goal to the generated FIT file if you generate the file from the segment details page

Land Registry data for January 2018

I’ve uploaded the latest property data for England and Wales to my website. Like a broken record, I can report that house prices continue their gradual rise ever upwards. Sales volumes are holding steady.

UK postcode data February 2018

I’ve uploaded the latest UK postcode data from the ONS to my website. Let me know if you spot any anomalies

Land Registry data December 2017

I’ve uploaded the latest house price data for England and Wales to my website. Not much to report, sales are static, prices continue to inch upwards

House price data for November 2017

I’ve uploaded the latest house price data from the Land Registry to my website. Prices continue their slow march upwards, although some areas of London have gone negative.