I’ve uploaded the latest data from the ONS to my website. I have run my usual sanity checks, let me know if you spot anything amiss
Tuesday, August 23, 2016
Wednesday, August 17, 2016
For many years I’ve been using the Google Maps APIs on my website. It’s been fun to use and until recently the licensing has been very unrestrictive. If an API returned a response saying you’d gone over the query limit, just wait for a second or so, try again and generally it would work. So with some use of setTimeout, it was possible to build reasonably scalable apps that cost nothing.
It’s looking like those days are coming to an end. The various APIs are starting to introduce hard limits on their usage. Once you’re over the limit, that’s it until the counter resets at the start of the next day. I first hit this with my use of the Directions API in my Driving Distances page. I can’t say I’m too happy with the way it was introduced, the Google API Console had given no previous indication of my usage of the API but on the same day as they started displaying the usage report, the hard limit was also introduced. Since my site was way over the limit, that page fell over almost immediately.
So I had a Baldrick cunning plan. I’d swap out the Directions API for the Distance Matrix API. This is exactly the kind of application this API was designed for. Unfortunately I failed to read the usage limits correctly and after I uploaded the new code, the page fell over in a heap again after a few hours. It turns out the usage limits apply to the elements passed to the Distance Matrix API, not the number of requests. So a 10 by 10 matrix counts as 100 towards the free 2,500 limit, not 1 as I had assumed. Given that this API provides less information and has fewer options than the Directions API but has the exact same usage limits, this is rather disappointing!
I am trying to figure out the best way forward now. I could start to pay for extra requests, but since a 100 by 100 matrix would cost $5, the costs could mount up quickly. I can put a maximum daily cost on the account so I don’t have to pay enormous amounts if someone overuses the page, but this could lead to the page becoming unavailable again.
I suspect the outcome will be me removing the page, or at least no longer linking to it from the rest of the site. I make a tidy sum from Google AdSense advertising on the site, but I fear this may just be the start, as more and more APIs start to introduce a hard limit and I don’t particularly want to pay that money back to Google every month to pay for their APIs. It was fun whilst it lasted I guess!
Saturday, July 09, 2016
So there’s a page on my site, https://www.doogal.co.uk/BatchGeocoding.php, that someone complained about. Specifically, they complained that if they tried to geocode 3,000 postcodes, it was terribly slow. I tried it myself and experienced the same problem. When geocoding postcodes, the page uses my own internal database, so it should suffer none of the throttling issues of Google Maps. No worries I thought, I can reproduce the problem, which is generally the biggest hurdle, fixing it should be straightforward.
So I fired up Chrome’s profiler and found… absolutely nothing… None of the delays were in my code. So I tried Microsoft Edge and it was super quick. I pretty much gave up at that point and suggested the user tried MS Edge.
Three weeks later I had a look with fresh eyes. And something popped up from the recesses of my mind, spellcheck=”false”. I vaguely remembered setting that attribute on a textarea in the past had improved performance and once again, this fixed the issue. A single geocode was previously taking a second, now all 3,000 took a couple of minutes. This may be a bug in Chrome or maybe spellchecking is a very CPU intensive process. Either way, turning it off makes everything better.
As always this is just a reminder for me and maybe it will be useful to someone passing through.
Friday, July 01, 2016
Tuesday, May 31, 2016
Wednesday, May 25, 2016
I’ve uploaded the latest UK postcode data to my website. Well, nearly all of it. Northern Irish postcodes are still from 2008 since the nice folk at NISRA still release their data under a restrictive license. I assumed at some point they would come into line with the rest of the UK and provide the postcode data with a liberal license, but it doesn’t seem to be happening. Since the data I have is now very old (from a time when the data was released with a liberal license), I am considering removing it from the site. Let me know if you find it useful and I’ll keep it online. And maybe send a polite request to NISRA to open up their data…