Wednesday, June 06, 2018

Goodbye Google Maps – important notice

A while back I received an email from Google Maps telling me they were introducing new pricing, but the email included the line “Based on your project usage over the last 3 months and our new pricing plan, we estimate that your new cost will be less than $200 a month and will be covered by our $200 monthly free credit” so I was pretty chuffed since I currently spend about £100 a month on Google Maps. I then promptly forgot about it.

But I then received another email saying the changes were being postponed and I thought it was time to look at how much I’d be paying. After running a few numbers, I realised I’ll have to pay several thousand pounds a month to continue to use Google Maps. Since this will wipe out any advertising revenue I make (via Google of course…), it looks like I’ll have to start removing and replacing various parts of the site.

The Google API Console is fairly useless. Although it tells me my API usage, it doesn’t tell me which pages are causing the usage so I don’t have a clear insight into what I need to remove. But two pages that are likely to be affected are the batch geocoding and batch reverse geocoding pages since they certainly generate a lot of requests. Other pages likely to be affected are random addresses, route elevation and driving distance. Other pages may fail to load maps or fail in other ways after July 16th. Apologies in advance

Wednesday, May 30, 2018

Land Registry house price data April 2018

I’ve uploaded the latest Land Registry house price data to my website. I’d read stories of annual house price falls but the Land Registry data is showing a 3% annual increase. The Land Registry numbers are a few months behind other house price indices so a fall may be on its way in a few months

Monday, May 28, 2018

One month later and TSB online banking is still a disaster

One month ago TSB tried to migrate their customers to a new online banking platform. Things didn’t go particularly well and although the media have lost interest in the problems, as a TSB customer and IT geek I’m still interested in what’s happening and thought I’d touch on a few of the issues.

The most glaring issue is that my mortgage account still doesn’t appear in my list of accounts. I assume I still have a mortgage since the payments are still being taken from my current account but there has been no sighting of it since the ‘upgrade’.

A few days ago, I tried to login, only to see an error message, <div>Sorry, we are currently experiencing technical issues. Please try again later.</div>. Not being able to login isn’t great but not totally unexpected given the recent troubles. Displaying HTML tags in your error messages is a fairly minor issue but suggests a lack of attention to detail that can lead to more serious issues down the road.

Now lets go under the hood by firing up Dev Tools in Chrome. The first error I saw was complaining about Symantec certificates which Chrome will stop supporting in a future release. Let’s hope someone at TSB renews that fairly soon before things break properly.

But the next set of errors are slightly more concerning, WebSocket connection to 'wss://127.0.0.1:3389/' failed, something on the web page is trying to connect to a server on my local machine! I can only assume this is some kind of debugging code but should definitely not be getting pushed out to their production system.

Next we have a failure to load a locale file for US English, which is odd since I’m set up for UK English. Again, nothing serious but shows a lack of attention to detail.

Finally, let’s run the page through Lighthouse, the Chrome web page checker. The performance score is a mighty 18 out of 100. This seems mostly due to Javascript and CSS that isn’t bundled together and isn’t being loaded using async or defer, meaning everything has to load before the page can be displayed.

And that’s what I found spending a few minutes on the TSB website, who knows what could be found if I was an evil hacker wanting to steal some money

Sunday, May 27, 2018

Converting KML file to shapefiles

For a long time I’ve been searching for some code that I can use on my website to convert the various KML postcode files that are available. Since many of these files are generated on the fly, I didn’t want to use a one off convertor. But I’ve always failed to find anything that will do the job. Since I don’t know the internals of shapefiles, it’s not something I wanted to write myself.

So I’ve kind of given up, but if you need to convert any of the KML files yourself, then this tool may be useful. From my admittedly not very thorough testing, it seems to do a good job of the conversion

Thursday, May 03, 2018

Land Registry house price data March 2018

I’ve uploaded the latest property data for England and Wales from the Land Registry to my website. Annual house price inflation has been slowly creeping up and now stands at 3%. Sales data is harder to get a handle on since it can take months for all the sales data to come through but it looks like sale may be dropping off a little.

Wednesday, March 28, 2018

Land Registry property sales data for February 2018

My server is still crunching through some of the numbers but most of the England and Wales house price data for February 2018 is up on my site. There’s very little to report, annual inflation remains at just short of 3%, number of transactions remains at a steady level

Thursday, March 08, 2018

My response to every anti-cycling article ever

If it’s not some Daily Mail article complaining about cyclists not using a cycle lane, it’s someone starting up a petition demanding cyclists should pay Road Tax (sorry no links but I don’t want to give them any encouragement). But one thing remains the same, the arguments put forward in the comments sections of these articles are always the same tired ill-thought out complaints. So rather than repeat myself on every article, here are my responses to the most common issues

Cyclists should pay road tax

The most common response to this is that road tax doesn’t exist, which is accurate, if a little pedantic. But we all presumably understand that Vehicle Excise Duty is charged based on the pollution caused by the vehicle, and I presume we can all agree the pollution generated by a bicycle is pretty much zero and hence the VED payable by the cyclist would be £0? OK, make us pay VED, I have no problem with that.

Beyond that, like most cyclists I own a car, for which I’m paying VED. I also pay Council Tax, Income Tax and VAT. I’ve just paid a big chunk of change in Stamp Duty to buy a house. Like most cyclists, I’m already contributing whilst helping to reduce congestion by using a bike instead of my car for every journey.

All cyclists jump red lights

Yes, some cyclists jump red lights, just like some drivers jump red lights. Just like some pedestrians do stupid things like crossing the road without looking. Cyclists are not an homogenous group. I suspect the problem here is that drivers feel that cyclists get away with jumping red lights. Here’s the thing, if a cyclist jumps a red light and causes an accident, the chances are he’ll come off worse or if he hits a pedestrian is unlikely to cause life changing injury. If someone in 2 tonnes of steel causes an accident, the chances of injury or death are much higher. So, with limited resources, the police are going to concentrate on the potentially more dangerous group of road users.

Cyclists should use the cycle lane when one is available

First up, there is no legal requirement for a cyclist to use a cycle lane if one is available. You can argue about whether that should be the case, but it is. But more importantly, a lot of cycle lanes are not fit for purpose. Take this example of road rage on Priory Lane in South West London. The driver gets insanely upset that the cyclist isn’t using the cycle lane (how stressful is driving BTW, get a bike, you’ll have much more fun…) but the cycle lane has a couple of problems. You have to give way at every junction along the road, of which there are quite a few. And when travelling South, it’s on the wrong side of the road. So if you are a confident cyclist, chances are you’ll use the road.

Or how about this 5 metre bit of cycle lane. Am I really expected to use that?

But these cycle lanes probably do work for some less confident cyclists and that is perhaps what drivers need to understand. Cycle lanes are not built for every cyclist, some are there to encourage more people to cycle, because that would be a good thing, right?

Cyclists should be required to have insurance

I’m almost in agreement with this. I’d have no problem getting insurance. But then what about the kid who has just got a shiny new bike for Christmas, they aren’t allowed to use it because their dad forgot to get insurance? Again, I am assuming encouraging cycling is the aim here

Cyclists should have number plates

Yeh, why not, if that floats your boat…

Saturday, March 03, 2018

Converting Strava segments to FIT files

Strava Premium users who own a Garmin can sync their favourite Strava segments with their Garmin device automatically. But that’s not an option available to free users. Apparently this site used to be able to generate the FIT files used by Garmins but Strava made some fairly radical changes to their API that broke that site (and many others).

So I’ve added a FIT file export to my Segment Explorer. Due to rather inclement weather I’ve not been able to test this out on the road yet, but downloading the FIT file and shoving it in my Garmin\NewFiles folder displays it in my Segments list so it seems to work. If you authorize my app then your PR time will also be included in the file.

Update – You can now add a goal to the generated FIT file if you generate the file from the segment details page

Wednesday, February 28, 2018

Land Registry data for January 2018

I’ve uploaded the latest property data for England and Wales to my website. Like a broken record, I can report that house prices continue their gradual rise ever upwards. Sales volumes are holding steady.

Wednesday, February 07, 2018

Wednesday, January 03, 2018

Sunday, November 26, 2017

Sunday, October 08, 2017

Adjusting Strava KOM speeds for gradient

I’ve been wanting to improve the algorithm behind my Strava segment chooser for some time. Until now it’s been based off the number of riders who’ve attempted the segment and the power output of the KOM. It gave reasonable results but had to ignore any segments whose KOM didn’t have any power data recorded (either from a power meter or estimated based off the rider’s weight and speed). And although more power means more speed, a heavier rider will have to output more power to go at the same speed so the power output isn’t the perfect indicator of how difficult a KOM is to beat.

But using speed also has its problems. 20 mph on a flat segment is generally not going to win any KOMs but 20mph on a 10% gradient may well do. Adjusting the KOM speed for the gradient of the segment has been my aim for a while, but I haven’t had a chance to look at it until now.

Since I now have quite a few segments in my database, I thought it was time to analyse KOM speeds against gradient. I put each segment in a bucket based on its gradient and calculated the average KOM speed. This gave me the following, rather surprising graph.

image

For gradients between about –2% and 10%, things work as I’d expect. The steeper the gradient, the slower the KOMs are. Beyond that, things get a bit weird. Steeper uphill segments get a little faster and steeper downhill segments get a little slower. I have a few ideas about why this might be (more mountain bike segments, the terror induced by a steep downhill) but nothing definitive.

Whatever the reason, I decided to ignore segments outside –2% to 10%. The graph in this range can be approximated to a straight line so calculating an adjusted speed is fairly easy. I’ve applied this to my segment chooser and the results are, er, much the same as before…

Friday, September 22, 2017

Convert a list to a comma string

It’s nothing special, but I’ve found this useful in the past and probably will again so here for my memory bank is a function that converts a list to a comma string


using System.Collections.Generic;
using System.Text;

namespace Utils
{
  public static class ListHelper
  {
    public static string ToCommaString<T>(this IEnumerable<T> list, string separator = ", ")
    {
      var builder = new StringBuilder();

      foreach (var t in list)
      {
        if (builder.Length > 0)
          builder.Append(separator);
        builder.Append(t);
      }

      return builder.ToString();
    }
  }
}

Friday, July 28, 2017

Wednesday, June 28, 2017

Land Registry data for May 2017

I’ve uploaded the latest Land Registry data for England and Wales to my website. My earlier predictions of house price growth going negative appear to be incorrect. Prices continue to go up but at an ever decreasing pace.

Monday, June 26, 2017

Recovering data from a crashed Garmin 810

I love my Garmin and it would be a perfect piece of tech if it weren’t for one problem. Almost every time I use it to follow a course, it will die at some point during the ride. I can turn it back on and after several minutes of confusion it will figure out where it is and I can resume recording. The suggestions on the Garmin forum mostly seem to be to downgrade to an older version of the firmware, but I’m too lazy to do that and keep hoping Garmin will fix it at some point.

But they haven’t yet, so I just live with it. So I thought I’d document what I do to recover the data from before the crash. When I plug the Garmin in, the data from after the crash will sync to Strava but it will also complain that something failed to sync. So the first thing to do is to manually upload the pre-crash data to Strava. This will be a .fit file in the Garmin\Activities folder of the device. Strava seems to be able to cope with the file, even though the Garmin sync can’t.

You may be happy enough just doing that, but I like to have an accurate record of my rides so I want to merge the two together. So I now

  • Export the two activities from Strava as GPX files.
  • Open them both in this tool.
  • Select both files, right click and merge them
  • In the newly created file, select both tracks, right click and select Restructuring/Join
  • Save the new file
  • Delete the two original activities from Strava
  • Upload the newly created file to Strava and feel satisfied with all the PRs I’ve hopefully achieved…

Wednesday, May 31, 2017

Property data for England and Wales April 2017

I’ve uploaded the April 2017 Land Registry data to my website. The annual change is still positive and although it’s heading towards zero, the progress is slowing.

Tuesday, May 30, 2017

Thursday, May 04, 2017

Land Registry house price data for March 2017

Another month passes by and we have a new batch of house price data from the Land Registry. The annual change remains positive but continues to creep downwards. In some areas, the annual change has gone negative, you can see more detailed figures here.

Tuesday, March 28, 2017

Land Registry house price data February 2017

The latest Land Registry sales data for England and Wales has been released and my server is currently churning through the data. The top level annual change remains positive but continues to trend downwards.

Thursday, March 02, 2017

Postcode data for February 2017

The ONS has released its latest postcode data, which is now on my site. I’ve run my usual sanity checks and it all seems to be in order, but let me know if you spot any anomalies.

Tuesday, February 28, 2017

Land Registry data for January 2017

Another month has flown by and it’s time for a house price update from the Land Registry. Sale volumes remain stable and prices continue to rise at an ever slower rate. It looks like the annual change will turn negative in 5 months if things continue in the same way.

Tuesday, January 31, 2017

Friday, December 30, 2016

Property sale data November 2016

I’ve uploaded the latest house price data from the Land Registry for England and Wales to my website. I’ve also made some changes to the way annual changes are calculated. This means the annual change figure is much less volatile, but it does mean changes in the directions of travel are slower to appear. The current annual change is 3.5% but reducing each month. If the current trend continues, the annual change will go negative in about 9 months time.

UK stations updated with 2016 usage data

I’ve uploaded the UK station usage data for 2016 to my site. I’ve also made a few minor tweaks to the layout which hopefully make the data more useful

Wednesday, November 30, 2016

Monday, November 28, 2016

Land Registry house price data for October 2016

I’ve uploaded the October 2016 Land Registry house price data to my website. Prices are just about managing to maintain a positive annual increase. Although houses are continuing their upward march, there seems to have been a mini collapse in the price of flats, I’m guessing due to the Buy To Let tax changes.

Friday, October 28, 2016

Land Registry data for September 2016

I’ve uploaded the latest house price data to my website. Prices keep creeping upwards, although there are lots of regional variations.

Friday, September 30, 2016

England and Wales house price data August 2016

As I type this, my server is churning through the latest Land Registry data for August 2016. The top level data has been imported and it suggests prices continue to glide ever upwards. The only difference seems to be the annual change is getting smaller, whereas it’s been around 5% for the last few years, it’s now around 2%.

If you look back at the data for earlier this year, you can now clearly see the spike in sales in March just before the buy to let Stamp Duty hike. Sales now seem to have returned to the levels seen after the financial crisis.

Wednesday, August 31, 2016

Tuesday, August 23, 2016

UK Postcode data August 2016

I’ve uploaded the latest data from the ONS to my website. I have run my usual sanity checks, let me know if you spot anything amiss

Wednesday, August 17, 2016

Google Maps Distance Matrix may not be what you’re after

For many years I’ve been using the Google Maps APIs on my website. It’s been fun to use and until recently the licensing has been very unrestrictive. If an API returned a response saying you’d gone over the query limit, just wait for a second or so, try again and generally it would work. So with some use of setTimeout, it was possible to build reasonably scalable apps that cost nothing.

It’s looking like those days are coming to an end. The various APIs are starting to introduce hard limits on their usage. Once you’re over the limit, that’s it until the counter resets at the start of the next day. I first hit this with my use of the Directions API in my Driving Distances page. I can’t say I’m too happy with the way it was introduced, the Google API Console had given no previous indication of my usage of the API but on the same day as they started displaying the usage report, the hard limit was also introduced. Since my site was way over the limit, that page fell over almost immediately.

So I had a Baldrick cunning plan. I’d swap out the Directions API for the Distance Matrix API. This is exactly the kind of application this API was designed for. Unfortunately I failed to read the usage limits correctly and after I uploaded the new code, the page fell over in a heap again after a few hours. It turns out the usage limits apply to the elements passed to the Distance Matrix API, not the number of requests. So a 10 by 10 matrix counts as 100 towards the free 2,500 limit, not 1 as I had assumed. Given that this API provides less information and has fewer options than the Directions API but has the exact same usage limits, this is rather disappointing!

I am trying to figure out the best way forward now. I could start to pay for extra requests, but since a 100 by 100 matrix would cost $5, the costs could mount up quickly. I can put a maximum daily cost on the account so I don’t have to pay enormous amounts if someone overuses the page, but this could lead to the page becoming unavailable again.

I suspect the outcome will be me removing the page, or at least no longer linking to it from the rest of the site. I make a tidy sum from Google AdSense advertising on the site, but I fear this may just be the start, as more and more APIs start to introduce a hard limit and I don’t particularly want to pay that money back to Google every month to pay for their APIs. It was fun whilst it lasted I guess!

Saturday, July 09, 2016

textareas slow in Chrome

So there’s a page on my site, https://www.doogal.co.uk/BatchGeocoding.php, that someone complained about. Specifically, they complained that if they tried to geocode 3,000 postcodes, it was terribly slow. I tried it myself and experienced the same problem. When geocoding postcodes, the page uses my own internal database, so it should suffer none of the throttling issues of Google Maps. No worries I thought, I can reproduce the problem, which is generally the biggest hurdle, fixing it should be straightforward.

So I fired up Chrome’s profiler and found… absolutely nothing… None of the delays were in my code. So I tried Microsoft Edge and it was super quick. I pretty much gave up at that point and suggested the user tried MS Edge.

Three weeks later I had a look with fresh eyes. And something popped up from the recesses of my mind, spellcheck=”false”. I vaguely remembered setting that attribute on a textarea in the past had improved performance and once again, this fixed the issue. A single geocode was previously taking a second, now all 3,000 took a couple of minutes. This may be a bug in Chrome or maybe spellchecking is a very CPU intensive process. Either way, turning it off makes everything better.

As always this is just a reminder for me and maybe it will be useful to someone passing through.

Friday, July 01, 2016

Tuesday, May 31, 2016

Land Registry sales data for April 2016

I’ve uploaded the latest Land Registry data to my site (although the postcode level summary data is still being generated). The annual change has dropped to 1%, which may well be the first effects of the BTL tax changes recently introduced.

Wednesday, May 25, 2016

UK postcode data for May 2016

I’ve uploaded the latest UK postcode data to my website. Well, nearly all of it. Northern Irish postcodes are still from 2008 since the nice folk at NISRA still release their data under a restrictive license. I assumed at some point they would come into line with the rest of the UK and provide the postcode data with a liberal license, but it doesn’t seem to be happening. Since the data I have is now very old (from a time when the data was released with a liberal license), I am considering removing it from the site. Let me know if you find it useful and I’ll keep it online. And maybe send a polite request to NISRA to open up their data…

Sunday, April 24, 2016

doogal.co.uk now using https

For a while now, Google has been trying to get everyone to move their sites over to https. There’s lots of valid reasons to do this, although the majority of sites don’t really need it.

The carrot of improved rankings hadn’t prompted me to make the change but Chrome 50 removed support for geolocation services which I use in a number of places. So the site was broken in Chrome 50. And it was that stick that motivated me to make the switch.

One reason I’d held off from using https was the cost of a certificate. But things have moved on and it’s now possible to grab a certificate for nothing from Let’s Encrypt. It’s pretty simple to acquire and install a certificate using these instructions

So the switch has been made. For most users of the site, nothing should have changed, unless I’ve broken something (let me know!). Comments are currently being migrated, so you may find some comments aren’t where they should be. If you are grabbing data from the site directly you may need to change the URL you use from http:// to https://. 

Sunday, April 10, 2016

Sampling Strava again

I thought I’d repeat the experiment conducted by Mark Slavonia last year to see how the Strava usage numbers stack up now.
First I needed to know the number of signed up users of Strava. This is pretty straightforward, head off to https://www.strava.com/athletes/6161562 and keep increasing the number at the end until Strava says it can’t find a user. Last March there were 8.2 million users, now there are about 14.4 million, not a bad increase for just over a year.
Next I wanted to capture the active users and the premium users. Since I’m a techy, I can automate this process using the Strava API and a .NET wrapper around it. So I decided to sample 1 in every 10,000 users, giving me about 1,440 sample users which should give the results a reasonable accuracy.
After pulling down that data, the first thing I noticed was that 47 of my requests had returned ‘Not Found’ errors. In fact, most of these were grouped together, suggesting Strava decided to restart their numbering with larger IDs at some point. So the total number of users is probably just shy of 14 million.

Premium Users

Of the 1.393 users I had left, 28 were Premium users, so approximately 2% of all users. This figure is pretty close to last year’s figure so I’m happy to believe it. That equates to 280,000 premium users or $16.6 million in revenue for Strava.
As an aside, I’m a Premium member, but not because Strava offers particularly compelling features for Premium users, but mainly to show my support for a website that is exceedingly useful and fun. I suspect Strava could differentiate between free and Premium a lot more to increase the percentage of Premium users. Take a look at all the functionality available at veloviewer.com.

Active Users

The Strava API doesn’t let me get activity data for other users, so I’m not able to find out how active users are directly. But it does provide an Updated field, which I’m hoping gets updated when a user uploads an activity (the Strava API docs are a little vague on this point). Using last year’s definition of an active user being someone who has done something in the last 24 days, how many active users are there? I found 181 users where that Updated field was in the last 24 days. That’s about 13%, or 1.8 million active users. The percentage is again fairly close to last year’s figure so I’m happy to go with it.

Gender

I found 806 men and 270 women and 317 blanks. Ignoring the blanks, that almost exactly a 75% / 25% split between men and women.

By Country

I think my sample is too small to draw accurate conclusions from the home countries of Strava users, but lets play with the numbers anyway. 463 had blank entries for the country which leaves 930 users with a country specified. I’ve removed countries with less than 5 users in my sample and then adjusted for population. Below I’ve highlighted the countries where more than 2% of the population have signed up for Strava. It seems like there is massive potential to increase usage in many countries, although that may depend on whether there is a culture of recreational running and riding in these countries (China and India being the obvious biggest potential markets). And little old blighty, the UK, is top of the pile. Go UK!
Country Population Sampled users Approximate Strava users % of population
Australia 24,309,330 35 527,000 2.2
Austria 8,569,633 5 75,000 0.9
Belgium 11,371,928 6 90,000 0.8
Brazil 209,567,920 74 1,114,000 0.5
Canada 36,286,378 25 376,000 1.0
Chile 18,131,850 6 90,000 0.5
Colombia 48,654,392 9 135,000 0.3
Denmark 5,690,750 5 75,000 1.3
France 64,668,129 35 527,000 0.8
Germany 80,682,351 18 271,000 0.3
India 1,326,801,576 5 75,000 0.0
Ireland 4,713,993 9 135,000 2.9
Italy 59,801,004 37 557,000 0.9
Japan 126,323,715 9 135,000 0.1
Mexico 128,632,004 6 90,000 0.0
Netherlands 16,979,729 32 482,000 2.8
New Zealand 4,565,185 9 135,000 3.0
Philippines 102,250,133 8 120,000 0.1
Poland 38,593,161 5 75,000 0.2
Portugal 10,304,434 16 241,000 2.3
Romania 19,372,734 6 90,000 0.5
Russia 143,439,832 12 181,000 0.1
South Africa 54,978,907 16 241,000 0.4
South Korea 50,503,933 14 211,000 0.4
Spain 46,064,604 46 692,000 1.5
Switzerland 8,379,477 7 105,000 1.3
Taiwan 23,395,600 14 211,000 0.9
Ukraine 44,624,373 6 90,000 0.2
UK 65,111,143 153 2,303,000 3.5
US 324,118,787 233 3,508,000 1.1

Wednesday, April 06, 2016

Strava segments by year

I thought I’d do a little analysis of the data I’m slowly collecting on my Strava segment explorer. Every segment has a numeric ID and these started at 1* and each new segment gets an ID of one greater than the most recent segment. So it’s quite easy to figure out how many segments are getting created over time.
So since I don’t have a huge dataset to analyse, lets see how many segments had been created at the end of every year.
2016-04-06 (1)
So what’s this tell us? It shows the total number of segments created at the end of every year and it looks like since 2011, the number of segments created every year has remained fairly constant. I guess the interesting question would be whether creation of segments can be used as some kind of proxy for usage of Strava since Strava keep this information confidential? I think the answer to that is probably no. A new user in an area already choc full of segments probably isn’t going to feel the need to create more, although they may create a few personal ones (home to work etc). Long term users probably already have all the segments they need. A better approach would be to repeat this study from last year.
But I guess it does show Strava is still being actively used by its users, beyond that it’s hard to say anything definitive.
*Not entirely true, the lowest ID I’ve found is 96, but I imagine the ID of the first segment ever created was 1.

Monday, February 29, 2016

UK postcodes with altitude data

After much faffing around, I have finally added altitude data to my postcode data. Altitudes should be included with all the various CSV downloads. Hopefully these are reasonably accurate, but let me know if you find anything obviously wrong.

Saturday, February 27, 2016

UK house price data January 2016

I’ve uploaded the latest Land Registry data to my site. The source file is a lot bigger than usual and seems to contain quite a lot of old sales.

Prices are rising even faster and the number of sales is on the increase.

Friday, February 26, 2016

February 2016 UK postcode data

I’ve uploaded the latest ONS postcode data to my website. I have run my usual sanity checks and it all seems OK but let me know if you spot any problems

Wednesday, February 03, 2016

Cycling speed variations with time of year

I’ve always assumed summer is the time of year when I will cycle the fastest. My ride data seems to back this up, with my average summer riding speed higher than my winter speed, but there are so many variables it is difficult to be definitive about it. For instance, during the summer I’m more likely to head out of the city and ride on some roads without too much traffic and fewer red lights, so I manage to maintain a higher average speed (although another variable to throw in is I tend to go up bigger hills where I’m slower!). Also I ride more during the summer so my overall fitness is probably higher. So maybe the difference in speed is not weather or temperature related.

But I was still surprised to read someone suggesting they go faster in the winter. The only time this happens to me is when I’m out during a windy winter day and catch a nice tailwind.

But I thought I’d check the Strava data from my site. Which months have the most KOMs? Checking that removes at least one variable, different roads. First I had to update my site to store the KOM date for each segment then I had to grab some data. I chose segments in the UK, removing those that were less than 0.5km (too easily messed up with dubious GPS data) and removing segments with less than 100 riders (not competitive enough) and this is what I got.

image

This is a fairly small sample but it certainly suggests the summer months are the best time to grab a KOM, which also suggests the summer is the fastest time of the year for cycling. But then a thought crossed my mind, people tend to cycle more during the summer months, so perhaps it’s unsurprising that most KOMs are achieved then. So my conclusion is that I still have no idea and more research is required. And there are too many variables…
For my own recollection, this is the SQL I used to grab the data
SELECT MONTH(KOMDate), COUNT(*) FROM segments
WHERE Country='United Kingdom' AND
Distance>500 and
TriesCount>100 and
KOMDate IS NOT NULL
GROUP BY MONTH(KOMDate)
ORDER BY MONTH(KOMDate)

Monday, February 01, 2016

Thursday, December 31, 2015

UK house prices November 2015

You can now view and download UK house price data for November 2015 from my website. Not much to report, prices continue on their ever upward trajectory, transactions appear to be creeping upwards

Tuesday, November 24, 2015

Postcode data for November 2015

I’ve uploaded the latest ONS postcode data for November 2015 to my website, all 2,554,806 of them. I’ve run my usual checks but let me know if you spot anything that looks incorrect.

Sunday, November 22, 2015

Fixing Strava elevation data

For some reason, Strava actually trust the data that comes from their users. More specifically they use the elevation data from the user’s ride when the user creates a segment. From a technical point of view, this is definitely the easiest thing to do, but unfortunately GPS devices do occasionally lose their mind, so the data can be a mess. This can lead to garbage segment data, like this. A glance at the elevation profile makes it obvious that something is amiss. This dodgy data then means any derived data is also dubious, such as the climb category and the VAM numbers. The KOM rider on this particular segment has a VAM of 9,992 which is over 5 times what a drugged up Lance Armstrong could achieve. Even my average VAM on category 4 climbs is over 1,000 which suggests I could make a good fist of keeping up with a bunch of professional cyclists. Which I couldn’t. Ever.
In an ideal world, Strava would fix up these dodgy segments in some way. One fix would be to average out all the elevation data from every rider who has ridden a segment. Alternatively, they could use the elevation data from one of the mapping services. Finally, they could make it easier to report bad data.
So whilst we wait for Strava to fix this issue, I thought I’d have a play with the second option. My Strava segment search tool now has the ability to view segments as well as view them on Strava. This is what the example segment looks like. It use Google Maps to calculate the elevation of the segment and adds that to the elevation profile, along with calculated statistics.

Monday, November 09, 2015

UK stations data

As a prelude to some other work I might one day get round to, I’ve uploaded a list of UK train stations to my website. It comes in CSV and KML flavours, with the KML highlighting the busiest stations (mostly in the South East, as if you need to ask).

Thursday, October 29, 2015

UK property sale data for September 2015

It’s the end of the month so it’s time to upload the latest Land Registry property data to my website. The data crunching is still in progress but I’m off on holiday for a few days so can’t wait for it to complete before posting here (calculating all the various averages can take quite some time). Predictably enough, the data shows house prices continuing their upward march.

One sale this month caught my eye. Flat 4, 19 Terrapin Road was the first flat we bought, back in 1999. It’s just changed hands again, for a cool £530,000. So in 16 years, the price has increased over fivefold… Just one example of the insanity of the London housing market.

Tuesday, September 29, 2015

UK house prices August 2015

I’ve uploaded the latest house sale data from the Land Registry to my website. Prices seems to be ticking up at an increasing rate, numbers of sales are not changing much. One would imagine without an increase in volumes, prices can’t remain at their current high level. but I could have said the same thing for the past 7 years…

Saturday, August 29, 2015

UK Property Sales July 2015

I have uploaded the latest Land Registry house price data for Jul 2015 to my site. Prices continue to rise moderately and sales continue to be at a low level

Wednesday, August 26, 2015

August 2015 UK postcode data

I’ve uploaded the latest UK postcode data to my website. It now contains 2,551,959 postcodes, including live and terminated postcodes. I’ve run sanity checks on the data and all appears well but let me know if you spot any problems.

Friday, August 07, 2015

How to top a Strava segment leaderboard

Strava is all about the segments and bragging rights are gained by being top of the leaderboard for a segment. But for those of us living in areas with many other cyclists, we are very unlikely to be fast enough to top most of the local segments, even if we get pushed along by a massive tailwind. Here’s an example near me, with nearly 50,000 riders attempting it over half a million times. The leaderboard contains a number of professional cyclists, since a number of races have passed through, so I’m never going to get anywhere near the top (since you asked, I’m at about 10,000 currently).
But we all want our own KOM/QOM, so what to do?

Find an obscure segment

Head over to my Strava segment search tool and zoom in and pan around a bit. You should see quite a few more segments than you’ll find with Strava’s own search. They’ll generally be less popular segments and hence more likely to have beatable times. 

Create your own obscure segment

I have a couple of KOMs for my rides to and from work. These are fairly meaningless KOMs since I’m the only person to ride one of the segments and the other one has only been ridden by one other rider. But if that’s enough to make you feel you’ve made it as a rider, then go ahead and create your own segment. For this to work, you need to decrease the chance of anyone else riding your segment, so stick to obscure roads, make the segment fairly long and choose a route that nobody would ever normally follow. This is a brilliant example, one day I’ll get round to riding it to see if I can top the leaderboard

Keep riding

A while back I headed out on a ride, going down some roads I haven’t explored before. On getting home I discovered I’d topped a leaderboard without even trying. Admittedly only 10 other people have ridden the segment but it still counts!

Cheat

There are websites that will take the output from your bike computer and shift it around so it appears you went faster than you did. No, really. It’s obvious chasing after Strava KOMs is a fairly pointless activity, but cheating to do it has to be the most ridiculous thing ever.
But what about inadvertently cheating? Whilst on a ride, my GPS went a bit haywire and for a few minutes I was at the top of a leaderboard. I guess the algorithms at Strava spotted the mistake (or the former holder of the KOM) and I was demoted pretty quickly. But what about this one? Everybody on the first page of that leaderboard is averaging over 78mph, which is very impressive for a hilly segment round Richmond Park. But if you look at the actual rides for those amazing times, none of them bear any relation to the actual segment, they are just in the same general area. Figure out how that bug works and you could be topping lots of leaderboards.

Thursday, July 30, 2015

Monday, July 13, 2015

Strava segment explorer

I’ve added a new page to my website that gives a new look at Strava segments. I’m not too keen on the segment explorer on the Strava site. It doesn’t let me zoom in fully so I can’t find obscure segments that a slow coach like me might have a chance of gaining a KOM. It also only ever displays 10 segments at a time so I can’t get a good feel of what’s around.
My explorer fixes these issues. I have a load of ideas on how to improve it, so let me know if you find it useful and what you’d like to see.

Saturday, June 27, 2015

UK property sales May 2015

I’ve uploaded the latest Land Registry house price data to my site. Prices are still slowly ticking up but it seems the mini-boom in sales of 2013 and 2014 is over. 

Wednesday, June 03, 2015

UK postcode data for May 2015

I have uploaded the latest ONS postcode data to my website. This means all the latest postcodes in England, Wales and Scotland are now available. Northern Ireland continues to go its own way with its licensing so I am unable to update the postcodes I have from 2008. If you’d like to see up to date Northern Irish postcodes, I suggest contacting NISRA and ask them to relax their licensing to be in line with the rest of the UK.

Tuesday, June 02, 2015

UK house prices April 2015

I’ve just uploaded the latest Land Registry data for April 2015 to my website. Nothing of note to report, prices continue to rise at a slow pace.

Saturday, May 30, 2015

Retrieving the most popular pages using Google Analytics API again…

I occasionally run a little app I wrote 4 years ago to grab the most popular pages on my website via Google Analytics and update the database so the website can display a top 10 list of those pages. I tried to run it this morning and it fell over in a heap. It seems that Google no longer supports the API I was using. Ho hum, shit happens, software does rust…

So time to drag out my old code. Except I couldn’t find it. So time to look at Google’s latest and greatest API and rebuild it from scratch. Here’s what’s required and a small code sample.

First download the Google API .NET libraries. There seem to be a whole host of Google API libraries in nuget, but for Google Analytics the following should get what you need and all the dependencies.

Install-Package Google.Apis.Analytics.v3

Then you’ll need to create a service account in the Google Developers Console. After creating this service account, create a P12 key for it and save it somewhere on your computer. Then add the service account email address to the Google Analytics account you want to access.

Next fire up Visual Studio and create a console application and add the following code

using Google.Apis.Analytics.v3;
using Google.Apis.Analytics.v3.Data;
using Google.Apis.Auth.OAuth2;
using Google.Apis.Services;
using System;
using System.Security.Cryptography.X509Certificates;

namespace UpdateTop10
{
  class Program
  {
    [STAThread]
    static void Main(string[] args)
    {
      try
      {
        Read();
      }
      catch (Exception ex)
      {
        Console.WriteLine("ERROR: " + ex.Message);
      }
      Console.WriteLine("Press any key to continue...");
      Console.ReadKey();
    }

    private static void Read()
    {
      String serviceAccountEmail = "<the service account email address>";

      var certificate = new X509Certificate2(@"<the location of your P12 key file>",
        "notasecret", X509KeyStorageFlags.Exportable);

      ServiceAccountCredential credential = new ServiceAccountCredential(
         new ServiceAccountCredential.Initializer(serviceAccountEmail)
         {
           Scopes = new[] { AnalyticsService.Scope.AnalyticsReadonly }
         }.FromCertificate(certificate));

      // Create the service.
      var service = new AnalyticsService(new BaseClientService.Initializer()
      {
        HttpClientInitializer = credential,
        ApplicationName = "<your application name in Developer Console>"
      });

      // Run the request.
      DataResource.GaResource.GetRequest req = service.Data.Ga.Get("ga:<the ID of the analytics view found under Admin/View Settings>", DateTime.Now.AddMonths(-1).ToString("yyyy-MM-dd"),
        DateTime.Now.ToString("yyyy-MM-dd"), "ga:visits");
      req.Dimensions = "ga:pagePath";
      req.Sort = "-ga:visits";
      GaData data = req.Execute();

      for (int i=0; i<20; i++)
      {
        Console.WriteLine(data.Rows[i][0] + ": " + data.Rows[i][1]);
      }
    }
  }
}

Fill in the bits between angled brackets with your details and give it a go. The top 20 visited pages from the last month should appear in your output.


Overall it wasn’t too painful, Some of the examples on the web seem to be written for older versions of the API which can cause some confusion and Google have such a huge number of APIs out there, finding the right one can be tricky, but once those hurdles are overcome, it’s reasonably straightforward.

Update - Windows Live Writer currently doesn't work with Blogger accounts, I'm guessing this is the same issue that I was having with my old little app. Hopefully Google told everyone they were turning off ClientLogin support, but it appears Microsoft didn't get the memo... And hopefully it gets fixed soon, because the Blogger editor is effing terrible

Monday, May 18, 2015

UK constituency and administrative area KML

A strange thing happened last week, visits to my site were affected by real world events. That’s the first time that has happened. The cause was the UK election and the pages affected by it were the pages devoted to UK electoral constituencies and in particular the constituency of South Thanet. After having a look at those pages, I came to the conclusion that any visitors may have been quite disappointed with the data available on those pages. So I’ve spent a bit of time improving them. They now include the area polygons of each constituency and also provide an estimate of the population and number of households in each constituency.

Whilst i was at it, I also added area polygons for administrative districts and wards. Enjoy!

Thursday, May 14, 2015

No trouble with counties

I wrote recently about the trouble I’ve had figuring out what to do about counties in the postcode data on my site. Thanks to a very helpful comment on that post, I finally figured what I hope is the correct course of action. Previously I only showed county information for postcodes that were located in a county council. The suggestion was to map postcodes to ceremonial counties, which I wasn’t really aware of before. But a quick look at Wikipedia suggested they was a pretty simple mapping between them and administrative areas, with the exception of a little complexity in Stockton-on-Tees. So that’s what I’ve done. You can now download postcode data for each English county here. Hopefully this meets all your county needs!  

Wednesday, May 06, 2015

Distance to sea, house price improvements

I made a few improvements to the website over the Bank Holiday weekend. Ever had a burning desire to know the distance from a postcode to the sea? Well now you can find out.

I’ve also updated the way individual house prices are listed so that all sales for a particular property are grouped together (example)

Saturday, May 02, 2015

UK house price data March 2015

I’ve uploaded the latest Land Registry house price data for March 2015 to my site. Prices continue to chug along at 4-5%, as they have been doing for the last 12 months. The CSV download data now includes a yearly summary and I’ve included charts for this annual data at the postcode district level (monthly data is too volatile to show anything meaningful in such small areas)

The trouble with counties

I include some county information with my UK postcode download data, but I get quite a few questions regarding it. The most common question is why don’t all the postcodes have an associated county. The answer is that county information is only shown for postcodes that are located in an administrative county council. So LA1 postcodes are listed as part of Lancashire but BB1 postcodes aren’t, even though most people would consider Blackburn to be in Lancashire. This page gives you an idea of how this works. Select ‘Counties’ in the dropdown and see all the gaps.

The second question that generally follows is whether I could include county information for each postcode. This is where things get tricky. Have a read of this Wikipedia page on the subject of counties. In short, there are the administrative county councils in use today, there are historical counties whose boundaries have changed many times and there are postal counties that used to be supplied by the Royal Mail.

So if I wanted to add county information for every postcode, my first decision would be which of these to use. The Royal Mail seem pretty keen to get rid of postal counties and the information is not provided with the freely available postcode data, so that’s not an option.

So another option would be to use historical counties. Leaving aside the fact these boundaries changed many times, I’m not sure using them would satisfy most users of my download data anyway. Taking an example from the referenced Wikipedia page, how many people would consider Brixton to be in Surrey?

And the final option would be to fill the county data from the local authority. So Brixton would now be in London, but BB1 would now be in ‘Blackburn with Darwen’. Now in this case it’s obvious that any postcodes in the ‘Blackburn with Darwen’ area should show Lancashire as the county so I could possibly map all these authorities to sensible counties, with some work. But even then, would this provide people with what they want? Kingston would be in London, although some people would expect it to be in Surrey, since Surrey county council is based there (and use Surrey in their address).

So in conclusion, counties are blooming tricky and I suspect no one size would fit all, hence the incomplete set of data on my site. If anyone has an opinion on whether there is a good approach to this, let me know.

Monday, March 30, 2015

Cycling halfway round the world

In July 2012 I started to use Endomondo to track my bike rides. Since then some things have changed in my cycling, I’ve moved from a mountain bike to a road bike, I’ve tried cycling 100 miles in one day (only to be thwarted by the weather)and I’ve started using Strava, but I’ve continued to use Endomondo. And today I reached the milestone of getting halfway round the world.

image

Last year I covered almost 6,000 miles. If I can continue at that level, then in a just over a couple of years I will have completed my virtual trip round the world. If you want to encourage me, then you can sponsor me in this year’s Ride London 100

Saturday, March 28, 2015

Loading Google Maps asynchronously

Google’s PageSpeed keeps telling me I should load scripts asynchronously to improve the performance of my website. Now in an ideal world I’d use something like RequireJS to implement this for all my scripts, but frankly that seems like a bit of a big task and more than likely I’d stuff it up and break large chunks of my web site. So I thought I’d start small and just load up Google Maps asynchronously. Google provide an example of how to do this, but I wanted to encapsulate that in a simple reusable function with a callback function parameter to run after the library had loaded. This is what I came up with in TypeScript.

function loadGoogleMaps(libraries: string, callback: () => void) {

  window["initGM"] = () => {
    callback();
  };

  var script = document.createElement("script");
  script.type = "text/javascript";
  var url: string = "http://maps.google.com/maps/api/js?callback=initGM";
  if (libraries != null && libraries !== "") {
    url += "&libraries=" + libraries;
  }
  script.src = url;
  document.body.appendChild(script);
}

UK house price data February 2015

I’ve uploaded the latest Land Registry data to my site. Prices continue on their seemingly never ending upward march.

Thursday, March 05, 2015

A cross browser XML parser

A post from three years ago detailing how to implement selectSingleNode for XML documents in a cross-browser friendly manner is still getting a good number of hits. Which I guess shows that developers still need to manipulate XML in browsers, even with the increasing popularity of JSON. When we started to rewrite our desktop app on the web, JSON seemed like the obvious choice, but we use XPath in a big way and we wanted our web app to be compatible with our desktop app, so we stuck with XML, which meant having to deal with the different ways XML is supported in different browsers. So we now have a reasonably well featured cross browser XML parser, the source of which you’ll find below.

A couple of things to note, this probably works in IE8 and below but I’ve never tested it since our app needs at least IE9. Also, the code is TypeScript rather than Javascript, since TypeScript is slightly less insane…

// stop TypeScript complaining about stuff we don't have definitions for
interface Window {
  DOMParser;
}
declare var XPathResult;

class XmlWrapper {
  private xmlDoc: any;
  constructor(xml: string) {
    try {
      // try Internet Explorer first. Although later versions have DOMParser, they don't implement evaluate
      this.xmlDoc = new ActiveXObject("Microsoft.XMLDOM");
      this.xmlDoc.async = false;
      this.xmlDoc.loadXML(xml);
      this.xmlDoc.setProperty("SelectionLanguage", "XPath");
    } catch (ex) {
      if (window.DOMParser) {
        var parser = new DOMParser();
        this.xmlDoc = parser.parseFromString(xml, "text/xml");
      } else {
        throw new Error("Can't find an XML parser!");
      }
    }
  }

  public selectSingleElement(xmlNode, elementPath: string): Element {
    return <Element>this.selectSingleNode(xmlNode, elementPath);
  }

  public selectSingleNode(xmlNode, elementPath: string): Node {
    if (xmlNode == null) {
      xmlNode = this.xmlDoc;
    }

    if (this.xmlDoc.evaluate) {
      var doc = xmlNode.ownerDocument;
      if (doc == null) {
        doc = xmlNode;
      }
      var nodes = doc.evaluate(elementPath, xmlNode, null, XPathResult.ANY_TYPE, null);
      var results = nodes.iterateNext();
      return results;
    } else {
      return xmlNode.selectSingleNode(elementPath);
    }
  }

  public selectElements(xmlNode, elementPath: string): Element[] {
    return <Element[]>this.selectNodes(xmlNode, elementPath);
  }

  public selectNodes(xmlNode, elementPath: string): Node[] {
    if (xmlNode == null) {
      xmlNode = this.xmlDoc;
    }

    if (this.xmlDoc.evaluate) {
      var doc = xmlNode.ownerDocument;
      if (doc == null) {
        doc = xmlNode;
      }

      var resultsArray = [];
      var results = doc.evaluate(elementPath, xmlNode, null, XPathResult.ANY_TYPE, null);
      var thisElement = results.iterateNext();
      while (thisElement) {
        resultsArray.push(thisElement);
        thisElement = results.iterateNext();
      }

      return resultsArray;
    } else {
      return xmlNode.selectNodes(elementPath);
    }
  }

  get documentElement(): Element {
    return <Element>(this.xmlDoc.documentElement);
  }

  public xml(): string {
    // for IE
    if (this.xmlDoc.documentElement.xml) {
      return this.xmlDoc.documentElement.xml;
    }

    // Chrome, FireFox
    if (this.xmlDoc.documentElement.outerHTML) {
      return this.xmlDoc.documentElement.outerHTML;
    }

    // Safari
    return (new XMLSerializer()).serializeToString(this.xmlDoc.documentElement);
  }

  public static getNodeText(node: Node): string {
    if (node == null) {
      return "";
    }

    var value: string = node.text;
    if (node.textContent) {
      value = node.textContent;
    }

    return value;
  }

  public static setNodeText(node: Node, value: string): void {
    if (node == null) {
      return;
    }

    if (node.text !== undefined) {
      node.text = value;
    }

    if (node.textContent !== undefined) {
      node.textContent = value;
    }
  }
} 

Sunday, March 01, 2015

Google Earth Pro is kind of free

I was excited when Google announced Google Earth Pro was going to become free. A lot of geographical data comes in Shape files, but these aren’t usable in any free applications I know of and can’t really be used on the web without converting to something like KML first. But Google Earth Pro can load up Shape files and convert them to KML/KMZ.

Downloading isn’t a problem but getting hold of a key doesn’t seem to work as far as I can tell. Following the link to grab a free key just keeps redirecting to the download page. So I gave up in the end and grabbed an illicit key instead. Not sure of the legality of that, but if it’s meant to be free and I still need a key and Google won’t give me one, what choice did I have? Anyway a search for “google earth pro serial key or number” gave me a link to a site that produced a key I could use.

Saturday, February 28, 2015

UK house price data January 2015

I have uploaded the latest Land Registry house price data to my site. Prices continue their gradual ascent.

The BBC reports the blindingly obvious that there is a wide gap in regional house prices. This is clearly visible if you compare my place of birth with my current abode. Blackburn, like many areas outside the South East, has seen pretty much static house prices since 2008. Kingston upon Thames, like most other areas in London, took a bit of a breather in 2008 and then continued on its upward trajectory. I guess the interesting question is whether this is a permanent change or if the differences in regional prices will revert back to their historical average at some point. I can’t pretend to know the answer to that, but I do know prices in London are insane…

Wednesday, February 25, 2015

UK Postcode data for February 2015

The ONS have released the latest version of their postcode dataset and I have now uploaded it to my website. A few sanity checks suggest it is OK but let me know if you spot anything strange

Sunday, February 08, 2015

How to beat a Strava PR on every ride

I think getting to, or maintaining, a healthy weight is fairly straightforward, in principle at least. Match your calorie intake with your calorie burning. Riders on the Tour de France eat 9000 calories a day, but don’t put on any weight for the obvious reason that they burn through all those calories. Eating less has never appealed to me, so the exercise side of the equation is the one I try to work on and it generally works OK for me.

But motivation can be a problem. Cycling is my exercise of choice and a winter of cold, wet and windy weather can rather reduce the will to get out on the road. I’ve found a few things to motivate me in the past, signing up with Endomondo (and trying to beat various personal bests), buying a new bike and training for the Ride London 100 to name a few.

Strava is the latest motivator. Every ride gives the opportunity to beat a PR on one segment or another, so there are many more chances to get a little boost from receiving a medal at the end of a ride. But sometimes, nothing. A ride of an hour may lead to no achievement.

But there is a way to almost guarantee a PR on every ride. First, sign up with veloviewer. This provides even more geeky information about every segment you’ve ridden. Next, filter the segment list for ones you’ve only ridden once or twice. Next, check the weather to find out the prevailing wind. Now find some segments where the wind will be behind you. Then plan your ride to include those segments. And finally, ride the route!

And voila, chances are you will beat one of your PRs on that ride. And even better, you’ll probably have ridden some new segments during your ride, which will now be in your list of potential PR segments.

I’ve been using this technique for the last few months and I’ve still got 140 segments within 5 miles of my house that I’ve ridden 2 or less times. Of course I’ll get bored of this at some point, but maybe I’ve already found my next motivation tool (aside from Ride London 2015), increasing my Eddington number

Saturday, January 31, 2015