Monday, June 26, 2017

Recovering data from a crashed Garmin 810

I love my Garmin and it would be a perfect piece of tech if it weren’t for one problem. Almost every time I use it to follow a course, it will die at some point during the ride. I can turn it back on and after several minutes of confusion it will figure out where it is and I can resume recording. The suggestions on the Garmin forum mostly seem to be to downgrade to an older version of the firmware, but I’m too lazy to do that and keep hoping Garmin will fix it at some point.

But they haven’t yet, so I just live with it. So I thought I’d document what I do to recover the data from before the crash. When I plug the Garmin in, the data from after the crash will sync to Strava but it will also complain that something failed to sync. So the first thing to do is to manually upload the pre-crash data to Strava. This will be a .fit file in the Garmin\Activities folder of the device. Strava seems to be able to cope with the file, even though the Garmin sync can’t.

You may be happy enough just doing that, but I like to have an accurate record of my rides so I want to merge the two together. So I now

  • Export the two activities from Strava as GPX files.
  • Open them both in this tool.
  • Select both files, right click and merge them
  • In the newly created file, select both tracks, right click and select Restructuring/Join
  • Save the new file
  • Delete the two original activities from Strava
  • Upload the newly created file to Strava and feel satisfied with all the PRs I’ve hopefully achieved…

Wednesday, May 31, 2017

Property data for England and Wales April 2017

I’ve uploaded the April 2017 Land Registry data to my website. The annual change is still positive and although it’s heading towards zero, the progress is slowing.

Tuesday, May 30, 2017

Thursday, May 04, 2017

Land Registry house price data for March 2017

Another month passes by and we have a new batch of house price data from the Land Registry. The annual change remains positive but continues to creep downwards. In some areas, the annual change has gone negative, you can see more detailed figures here.

Tuesday, March 28, 2017

Land Registry house price data February 2017

The latest Land Registry sales data for England and Wales has been released and my server is currently churning through the data. The top level annual change remains positive but continues to trend downwards.

Thursday, March 02, 2017

Postcode data for February 2017

The ONS has released its latest postcode data, which is now on my site. I’ve run my usual sanity checks and it all seems to be in order, but let me know if you spot any anomalies.

Tuesday, February 28, 2017

Land Registry data for January 2017

Another month has flown by and it’s time for a house price update from the Land Registry. Sale volumes remain stable and prices continue to rise at an ever slower rate. It looks like the annual change will turn negative in 5 months if things continue in the same way.

Tuesday, January 31, 2017

Friday, December 30, 2016

Property sale data November 2016

I’ve uploaded the latest house price data from the Land Registry for England and Wales to my website. I’ve also made some changes to the way annual changes are calculated. This means the annual change figure is much less volatile, but it does mean changes in the directions of travel are slower to appear. The current annual change is 3.5% but reducing each month. If the current trend continues, the annual change will go negative in about 9 months time.

UK stations updated with 2016 usage data

I’ve uploaded the UK station usage data for 2016 to my site. I’ve also made a few minor tweaks to the layout which hopefully make the data more useful

Wednesday, November 30, 2016

Monday, November 28, 2016

Land Registry house price data for October 2016

I’ve uploaded the October 2016 Land Registry house price data to my website. Prices are just about managing to maintain a positive annual increase. Although houses are continuing their upward march, there seems to have been a mini collapse in the price of flats, I’m guessing due to the Buy To Let tax changes.

Friday, October 28, 2016

Land Registry data for September 2016

I’ve uploaded the latest house price data to my website. Prices keep creeping upwards, although there are lots of regional variations.

Friday, September 30, 2016

England and Wales house price data August 2016

As I type this, my server is churning through the latest Land Registry data for August 2016. The top level data has been imported and it suggests prices continue to glide ever upwards. The only difference seems to be the annual change is getting smaller, whereas it’s been around 5% for the last few years, it’s now around 2%.

If you look back at the data for earlier this year, you can now clearly see the spike in sales in March just before the buy to let Stamp Duty hike. Sales now seem to have returned to the levels seen after the financial crisis.

Wednesday, August 31, 2016

Tuesday, August 23, 2016

UK Postcode data August 2016

I’ve uploaded the latest data from the ONS to my website. I have run my usual sanity checks, let me know if you spot anything amiss

Wednesday, August 17, 2016

Google Maps Distance Matrix may not be what you’re after

For many years I’ve been using the Google Maps APIs on my website. It’s been fun to use and until recently the licensing has been very unrestrictive. If an API returned a response saying you’d gone over the query limit, just wait for a second or so, try again and generally it would work. So with some use of setTimeout, it was possible to build reasonably scalable apps that cost nothing.

It’s looking like those days are coming to an end. The various APIs are starting to introduce hard limits on their usage. Once you’re over the limit, that’s it until the counter resets at the start of the next day. I first hit this with my use of the Directions API in my Driving Distances page. I can’t say I’m too happy with the way it was introduced, the Google API Console had given no previous indication of my usage of the API but on the same day as they started displaying the usage report, the hard limit was also introduced. Since my site was way over the limit, that page fell over almost immediately.

So I had a Baldrick cunning plan. I’d swap out the Directions API for the Distance Matrix API. This is exactly the kind of application this API was designed for. Unfortunately I failed to read the usage limits correctly and after I uploaded the new code, the page fell over in a heap again after a few hours. It turns out the usage limits apply to the elements passed to the Distance Matrix API, not the number of requests. So a 10 by 10 matrix counts as 100 towards the free 2,500 limit, not 1 as I had assumed. Given that this API provides less information and has fewer options than the Directions API but has the exact same usage limits, this is rather disappointing!

I am trying to figure out the best way forward now. I could start to pay for extra requests, but since a 100 by 100 matrix would cost $5, the costs could mount up quickly. I can put a maximum daily cost on the account so I don’t have to pay enormous amounts if someone overuses the page, but this could lead to the page becoming unavailable again.

I suspect the outcome will be me removing the page, or at least no longer linking to it from the rest of the site. I make a tidy sum from Google AdSense advertising on the site, but I fear this may just be the start, as more and more APIs start to introduce a hard limit and I don’t particularly want to pay that money back to Google every month to pay for their APIs. It was fun whilst it lasted I guess!

Saturday, July 09, 2016

textareas slow in Chrome

So there’s a page on my site, https://www.doogal.co.uk/BatchGeocoding.php, that someone complained about. Specifically, they complained that if they tried to geocode 3,000 postcodes, it was terribly slow. I tried it myself and experienced the same problem. When geocoding postcodes, the page uses my own internal database, so it should suffer none of the throttling issues of Google Maps. No worries I thought, I can reproduce the problem, which is generally the biggest hurdle, fixing it should be straightforward.

So I fired up Chrome’s profiler and found… absolutely nothing… None of the delays were in my code. So I tried Microsoft Edge and it was super quick. I pretty much gave up at that point and suggested the user tried MS Edge.

Three weeks later I had a look with fresh eyes. And something popped up from the recesses of my mind, spellcheck=”false”. I vaguely remembered setting that attribute on a textarea in the past had improved performance and once again, this fixed the issue. A single geocode was previously taking a second, now all 3,000 took a couple of minutes. This may be a bug in Chrome or maybe spellchecking is a very CPU intensive process. Either way, turning it off makes everything better.

As always this is just a reminder for me and maybe it will be useful to someone passing through.

Friday, July 01, 2016

Tuesday, May 31, 2016

Land Registry sales data for April 2016

I’ve uploaded the latest Land Registry data to my site (although the postcode level summary data is still being generated). The annual change has dropped to 1%, which may well be the first effects of the BTL tax changes recently introduced.

Wednesday, May 25, 2016

UK postcode data for May 2016

I’ve uploaded the latest UK postcode data to my website. Well, nearly all of it. Northern Irish postcodes are still from 2008 since the nice folk at NISRA still release their data under a restrictive license. I assumed at some point they would come into line with the rest of the UK and provide the postcode data with a liberal license, but it doesn’t seem to be happening. Since the data I have is now very old (from a time when the data was released with a liberal license), I am considering removing it from the site. Let me know if you find it useful and I’ll keep it online. And maybe send a polite request to NISRA to open up their data…

Sunday, April 24, 2016

doogal.co.uk now using https

For a while now, Google has been trying to get everyone to move their sites over to https. There’s lots of valid reasons to do this, although the majority of sites don’t really need it.

The carrot of improved rankings hadn’t prompted me to make the change but Chrome 50 removed support for geolocation services which I use in a number of places. So the site was broken in Chrome 50. And it was that stick that motivated me to make the switch.

One reason I’d held off from using https was the cost of a certificate. But things have moved on and it’s now possible to grab a certificate for nothing from Let’s Encrypt. It’s pretty simple to acquire and install a certificate using these instructions

So the switch has been made. For most users of the site, nothing should have changed, unless I’ve broken something (let me know!). Comments are currently being migrated, so you may find some comments aren’t where they should be. If you are grabbing data from the site directly you may need to change the URL you use from http:// to https://. 

Sunday, April 10, 2016

Sampling Strava again

I thought I’d repeat the experiment conducted by Mark Slavonia last year to see how the Strava usage numbers stack up now.
First I needed to know the number of signed up users of Strava. This is pretty straightforward, head off to https://www.strava.com/athletes/6161562 and keep increasing the number at the end until Strava says it can’t find a user. Last March there were 8.2 million users, now there are about 14.4 million, not a bad increase for just over a year.
Next I wanted to capture the active users and the premium users. Since I’m a techy, I can automate this process using the Strava API and a .NET wrapper around it. So I decided to sample 1 in every 10,000 users, giving me about 1,440 sample users which should give the results a reasonable accuracy.
After pulling down that data, the first thing I noticed was that 47 of my requests had returned ‘Not Found’ errors. In fact, most of these were grouped together, suggesting Strava decided to restart their numbering with larger IDs at some point. So the total number of users is probably just shy of 14 million.

Premium Users

Of the 1.393 users I had left, 28 were Premium users, so approximately 2% of all users. This figure is pretty close to last year’s figure so I’m happy to believe it. That equates to 280,000 premium users or $16.6 million in revenue for Strava.
As an aside, I’m a Premium member, but not because Strava offers particularly compelling features for Premium users, but mainly to show my support for a website that is exceedingly useful and fun. I suspect Strava could differentiate between free and Premium a lot more to increase the percentage of Premium users. Take a look at all the functionality available at veloviewer.com.

Active Users

The Strava API doesn’t let me get activity data for other users, so I’m not able to find out how active users are directly. But it does provide an Updated field, which I’m hoping gets updated when a user uploads an activity (the Strava API docs are a little vague on this point). Using last year’s definition of an active user being someone who has done something in the last 24 days, how many active users are there? I found 181 users where that Updated field was in the last 24 days. That’s about 13%, or 1.8 million active users. The percentage is again fairly close to last year’s figure so I’m happy to go with it.

Gender

I found 806 men and 270 women and 317 blanks. Ignoring the blanks, that almost exactly a 75% / 25% split between men and women.

By Country

I think my sample is too small to draw accurate conclusions from the home countries of Strava users, but lets play with the numbers anyway. 463 had blank entries for the country which leaves 930 users with a country specified. I’ve removed countries with less than 5 users in my sample and then adjusted for population. Below I’ve highlighted the countries where more than 2% of the population have signed up for Strava. It seems like there is massive potential to increase usage in many countries, although that may depend on whether there is a culture of recreational running and riding in these countries (China and India being the obvious biggest potential markets). And little old blighty, the UK, is top of the pile. Go UK!
Country Population Sampled users Approximate Strava users % of population
Australia 24,309,330 35 527,000 2.2
Austria 8,569,633 5 75,000 0.9
Belgium 11,371,928 6 90,000 0.8
Brazil 209,567,920 74 1,114,000 0.5
Canada 36,286,378 25 376,000 1.0
Chile 18,131,850 6 90,000 0.5
Colombia 48,654,392 9 135,000 0.3
Denmark 5,690,750 5 75,000 1.3
France 64,668,129 35 527,000 0.8
Germany 80,682,351 18 271,000 0.3
India 1,326,801,576 5 75,000 0.0
Ireland 4,713,993 9 135,000 2.9
Italy 59,801,004 37 557,000 0.9
Japan 126,323,715 9 135,000 0.1
Mexico 128,632,004 6 90,000 0.0
Netherlands 16,979,729 32 482,000 2.8
New Zealand 4,565,185 9 135,000 3.0
Philippines 102,250,133 8 120,000 0.1
Poland 38,593,161 5 75,000 0.2
Portugal 10,304,434 16 241,000 2.3
Romania 19,372,734 6 90,000 0.5
Russia 143,439,832 12 181,000 0.1
South Africa 54,978,907 16 241,000 0.4
South Korea 50,503,933 14 211,000 0.4
Spain 46,064,604 46 692,000 1.5
Switzerland 8,379,477 7 105,000 1.3
Taiwan 23,395,600 14 211,000 0.9
Ukraine 44,624,373 6 90,000 0.2
UK 65,111,143 153 2,303,000 3.5
US 324,118,787 233 3,508,000 1.1

Wednesday, April 06, 2016

Strava segments by year

I thought I’d do a little analysis of the data I’m slowly collecting on my Strava segment explorer. Every segment has a numeric ID and these started at 1* and each new segment gets an ID of one greater than the most recent segment. So it’s quite easy to figure out how many segments are getting created over time.
So since I don’t have a huge dataset to analyse, lets see how many segments had been created at the end of every year.
2016-04-06 (1)
So what’s this tell us? It shows the total number of segments created at the end of every year and it looks like since 2011, the number of segments created every year has remained fairly constant. I guess the interesting question would be whether creation of segments can be used as some kind of proxy for usage of Strava since Strava keep this information confidential? I think the answer to that is probably no. A new user in an area already choc full of segments probably isn’t going to feel the need to create more, although they may create a few personal ones (home to work etc). Long term users probably already have all the segments they need. A better approach would be to repeat this study from last year.
But I guess it does show Strava is still being actively used by its users, beyond that it’s hard to say anything definitive.
*Not entirely true, the lowest ID I’ve found is 96, but I imagine the ID of the first segment ever created was 1.

Monday, February 29, 2016

UK postcodes with altitude data

After much faffing around, I have finally added altitude data to my postcode data. Altitudes should be included with all the various CSV downloads. Hopefully these are reasonably accurate, but let me know if you find anything obviously wrong.

Saturday, February 27, 2016

UK house price data January 2016

I’ve uploaded the latest Land Registry data to my site. The source file is a lot bigger than usual and seems to contain quite a lot of old sales.

Prices are rising even faster and the number of sales is on the increase.

Friday, February 26, 2016

February 2016 UK postcode data

I’ve uploaded the latest ONS postcode data to my website. I have run my usual sanity checks and it all seems OK but let me know if you spot any problems

Wednesday, February 03, 2016

Cycling speed variations with time of year

I’ve always assumed summer is the time of year when I will cycle the fastest. My ride data seems to back this up, with my average summer riding speed higher than my winter speed, but there are so many variables it is difficult to be definitive about it. For instance, during the summer I’m more likely to head out of the city and ride on some roads without too much traffic and fewer red lights, so I manage to maintain a higher average speed (although another variable to throw in is I tend to go up bigger hills where I’m slower!). Also I ride more during the summer so my overall fitness is probably higher. So maybe the difference in speed is not weather or temperature related.

But I was still surprised to read someone suggesting they go faster in the winter. The only time this happens to me is when I’m out during a windy winter day and catch a nice tailwind.

But I thought I’d check the Strava data from my site. Which months have the most KOMs? Checking that removes at least one variable, different roads. First I had to update my site to store the KOM date for each segment then I had to grab some data. I chose segments in the UK, removing those that were less than 0.5km (too easily messed up with dubious GPS data) and removing segments with less than 100 riders (not competitive enough) and this is what I got.

image

This is a fairly small sample but it certainly suggests the summer months are the best time to grab a KOM, which also suggests the summer is the fastest time of the year for cycling. But then a thought crossed my mind, people tend to cycle more during the summer months, so perhaps it’s unsurprising that most KOMs are achieved then. So my conclusion is that I still have no idea and more research is required. And there are too many variables…
For my own recollection, this is the SQL I used to grab the data
SELECT MONTH(KOMDate), COUNT(*) FROM segments
WHERE Country='United Kingdom' AND
Distance>500 and
TriesCount>100 and
KOMDate IS NOT NULL
GROUP BY MONTH(KOMDate)
ORDER BY MONTH(KOMDate)

Monday, February 01, 2016

Thursday, December 31, 2015

UK house prices November 2015

You can now view and download UK house price data for November 2015 from my website. Not much to report, prices continue on their ever upward trajectory, transactions appear to be creeping upwards

Tuesday, November 24, 2015

Postcode data for November 2015

I’ve uploaded the latest ONS postcode data for November 2015 to my website, all 2,554,806 of them. I’ve run my usual checks but let me know if you spot anything that looks incorrect.

Sunday, November 22, 2015

Fixing Strava elevation data

For some reason, Strava actually trust the data that comes from their users. More specifically they use the elevation data from the user’s ride when the user creates a segment. From a technical point of view, this is definitely the easiest thing to do, but unfortunately GPS devices do occasionally lose their mind, so the data can be a mess. This can lead to garbage segment data, like this. A glance at the elevation profile makes it obvious that something is amiss. This dodgy data then means any derived data is also dubious, such as the climb category and the VAM numbers. The KOM rider on this particular segment has a VAM of 9,992 which is over 5 times what a drugged up Lance Armstrong could achieve. Even my average VAM on category 4 climbs is over 1,000 which suggests I could make a good fist of keeping up with a bunch of professional cyclists. Which I couldn’t. Ever.
In an ideal world, Strava would fix up these dodgy segments in some way. One fix would be to average out all the elevation data from every rider who has ridden a segment. Alternatively, they could use the elevation data from one of the mapping services. Finally, they could make it easier to report bad data.
So whilst we wait for Strava to fix this issue, I thought I’d have a play with the second option. My Strava segment search tool now has the ability to view segments as well as view them on Strava. This is what the example segment looks like. It use Google Maps to calculate the elevation of the segment and adds that to the elevation profile, along with calculated statistics.

Monday, November 09, 2015

UK stations data

As a prelude to some other work I might one day get round to, I’ve uploaded a list of UK train stations to my website. It comes in CSV and KML flavours, with the KML highlighting the busiest stations (mostly in the South East, as if you need to ask).

Thursday, October 29, 2015

UK property sale data for September 2015

It’s the end of the month so it’s time to upload the latest Land Registry property data to my website. The data crunching is still in progress but I’m off on holiday for a few days so can’t wait for it to complete before posting here (calculating all the various averages can take quite some time). Predictably enough, the data shows house prices continuing their upward march.

One sale this month caught my eye. Flat 4, 19 Terrapin Road was the first flat we bought, back in 1999. It’s just changed hands again, for a cool £530,000. So in 16 years, the price has increased over fivefold… Just one example of the insanity of the London housing market.

Tuesday, September 29, 2015

UK house prices August 2015

I’ve uploaded the latest house sale data from the Land Registry to my website. Prices seems to be ticking up at an increasing rate, numbers of sales are not changing much. One would imagine without an increase in volumes, prices can’t remain at their current high level. but I could have said the same thing for the past 7 years…

Saturday, August 29, 2015

UK Property Sales July 2015

I have uploaded the latest Land Registry house price data for Jul 2015 to my site. Prices continue to rise moderately and sales continue to be at a low level

Wednesday, August 26, 2015

August 2015 UK postcode data

I’ve uploaded the latest UK postcode data to my website. It now contains 2,551,959 postcodes, including live and terminated postcodes. I’ve run sanity checks on the data and all appears well but let me know if you spot any problems.

Friday, August 07, 2015

How to top a Strava segment leaderboard

Strava is all about the segments and bragging rights are gained by being top of the leaderboard for a segment. But for those of us living in areas with many other cyclists, we are very unlikely to be fast enough to top most of the local segments, even if we get pushed along by a massive tailwind. Here’s an example near me, with nearly 50,000 riders attempting it over half a million times. The leaderboard contains a number of professional cyclists, since a number of races have passed through, so I’m never going to get anywhere near the top (since you asked, I’m at about 10,000 currently).
But we all want our own KOM/QOM, so what to do?

Find an obscure segment

Head over to my Strava segment search tool and zoom in and pan around a bit. You should see quite a few more segments than you’ll find with Strava’s own search. They’ll generally be less popular segments and hence more likely to have beatable times. 

Create your own obscure segment

I have a couple of KOMs for my rides to and from work. These are fairly meaningless KOMs since I’m the only person to ride one of the segments and the other one has only been ridden by one other rider. But if that’s enough to make you feel you’ve made it as a rider, then go ahead and create your own segment. For this to work, you need to decrease the chance of anyone else riding your segment, so stick to obscure roads, make the segment fairly long and choose a route that nobody would ever normally follow. This is a brilliant example, one day I’ll get round to riding it to see if I can top the leaderboard

Keep riding

A while back I headed out on a ride, going down some roads I haven’t explored before. On getting home I discovered I’d topped a leaderboard without even trying. Admittedly only 10 other people have ridden the segment but it still counts!

Cheat

There are websites that will take the output from your bike computer and shift it around so it appears you went faster than you did. No, really. It’s obvious chasing after Strava KOMs is a fairly pointless activity, but cheating to do it has to be the most ridiculous thing ever.
But what about inadvertently cheating? Whilst on a ride, my GPS went a bit haywire and for a few minutes I was at the top of a leaderboard. I guess the algorithms at Strava spotted the mistake (or the former holder of the KOM) and I was demoted pretty quickly. But what about this one? Everybody on the first page of that leaderboard is averaging over 78mph, which is very impressive for a hilly segment round Richmond Park. But if you look at the actual rides for those amazing times, none of them bear any relation to the actual segment, they are just in the same general area. Figure out how that bug works and you could be topping lots of leaderboards.

Thursday, July 30, 2015

Monday, July 13, 2015

Strava segment explorer

I’ve added a new page to my website that gives a new look at Strava segments. I’m not too keen on the segment explorer on the Strava site. It doesn’t let me zoom in fully so I can’t find obscure segments that a slow coach like me might have a chance of gaining a KOM. It also only ever displays 10 segments at a time so I can’t get a good feel of what’s around.
My explorer fixes these issues. I have a load of ideas on how to improve it, so let me know if you find it useful and what you’d like to see.

Saturday, June 27, 2015

UK property sales May 2015

I’ve uploaded the latest Land Registry house price data to my site. Prices are still slowly ticking up but it seems the mini-boom in sales of 2013 and 2014 is over. 

Wednesday, June 03, 2015

UK postcode data for May 2015

I have uploaded the latest ONS postcode data to my website. This means all the latest postcodes in England, Wales and Scotland are now available. Northern Ireland continues to go its own way with its licensing so I am unable to update the postcodes I have from 2008. If you’d like to see up to date Northern Irish postcodes, I suggest contacting NISRA and ask them to relax their licensing to be in line with the rest of the UK.

Tuesday, June 02, 2015

UK house prices April 2015

I’ve just uploaded the latest Land Registry data for April 2015 to my website. Nothing of note to report, prices continue to rise at a slow pace.

Saturday, May 30, 2015

Retrieving the most popular pages using Google Analytics API again…

I occasionally run a little app I wrote 4 years ago to grab the most popular pages on my website via Google Analytics and update the database so the website can display a top 10 list of those pages. I tried to run it this morning and it fell over in a heap. It seems that Google no longer supports the API I was using. Ho hum, shit happens, software does rust…

So time to drag out my old code. Except I couldn’t find it. So time to look at Google’s latest and greatest API and rebuild it from scratch. Here’s what’s required and a small code sample.

First download the Google API .NET libraries. There seem to be a whole host of Google API libraries in nuget, but for Google Analytics the following should get what you need and all the dependencies.

Install-Package Google.Apis.Analytics.v3

Then you’ll need to create a service account in the Google Developers Console. After creating this service account, create a P12 key for it and save it somewhere on your computer. Then add the service account email address to the Google Analytics account you want to access.

Next fire up Visual Studio and create a console application and add the following code

using Google.Apis.Analytics.v3;
using Google.Apis.Analytics.v3.Data;
using Google.Apis.Auth.OAuth2;
using Google.Apis.Services;
using System;
using System.Security.Cryptography.X509Certificates;

namespace UpdateTop10
{
  class Program
  {
    [STAThread]
    static void Main(string[] args)
    {
      try
      {
        Read();
      }
      catch (Exception ex)
      {
        Console.WriteLine("ERROR: " + ex.Message);
      }
      Console.WriteLine("Press any key to continue...");
      Console.ReadKey();
    }

    private static void Read()
    {
      String serviceAccountEmail = "<the service account email address>";

      var certificate = new X509Certificate2(@"<the location of your P12 key file>",
        "notasecret", X509KeyStorageFlags.Exportable);

      ServiceAccountCredential credential = new ServiceAccountCredential(
         new ServiceAccountCredential.Initializer(serviceAccountEmail)
         {
           Scopes = new[] { AnalyticsService.Scope.AnalyticsReadonly }
         }.FromCertificate(certificate));

      // Create the service.
      var service = new AnalyticsService(new BaseClientService.Initializer()
      {
        HttpClientInitializer = credential,
        ApplicationName = "<your application name in Developer Console>"
      });

      // Run the request.
      DataResource.GaResource.GetRequest req = service.Data.Ga.Get("ga:<the ID of the analytics view found under Admin/View Settings>", DateTime.Now.AddMonths(-1).ToString("yyyy-MM-dd"),
        DateTime.Now.ToString("yyyy-MM-dd"), "ga:visits");
      req.Dimensions = "ga:pagePath";
      req.Sort = "-ga:visits";
      GaData data = req.Execute();

      for (int i=0; i<20; i++)
      {
        Console.WriteLine(data.Rows[i][0] + ": " + data.Rows[i][1]);
      }
    }
  }
}

Fill in the bits between angled brackets with your details and give it a go. The top 20 visited pages from the last month should appear in your output.


Overall it wasn’t too painful, Some of the examples on the web seem to be written for older versions of the API which can cause some confusion and Google have such a huge number of APIs out there, finding the right one can be tricky, but once those hurdles are overcome, it’s reasonably straightforward.

Update - Windows Live Writer currently doesn't work with Blogger accounts, I'm guessing this is the same issue that I was having with my old little app. Hopefully Google told everyone they were turning off ClientLogin support, but it appears Microsoft didn't get the memo... And hopefully it gets fixed soon, because the Blogger editor is effing terrible

Monday, May 18, 2015

UK constituency and administrative area KML

A strange thing happened last week, visits to my site were affected by real world events. That’s the first time that has happened. The cause was the UK election and the pages affected by it were the pages devoted to UK electoral constituencies and in particular the constituency of South Thanet. After having a look at those pages, I came to the conclusion that any visitors may have been quite disappointed with the data available on those pages. So I’ve spent a bit of time improving them. They now include the area polygons of each constituency and also provide an estimate of the population and number of households in each constituency.

Whilst i was at it, I also added area polygons for administrative districts and wards. Enjoy!

Thursday, May 14, 2015

No trouble with counties

I wrote recently about the trouble I’ve had figuring out what to do about counties in the postcode data on my site. Thanks to a very helpful comment on that post, I finally figured what I hope is the correct course of action. Previously I only showed county information for postcodes that were located in a county council. The suggestion was to map postcodes to ceremonial counties, which I wasn’t really aware of before. But a quick look at Wikipedia suggested they was a pretty simple mapping between them and administrative areas, with the exception of a little complexity in Stockton-on-Tees. So that’s what I’ve done. You can now download postcode data for each English county here. Hopefully this meets all your county needs!  

Wednesday, May 06, 2015

Distance to sea, house price improvements

I made a few improvements to the website over the Bank Holiday weekend. Ever had a burning desire to know the distance from a postcode to the sea? Well now you can find out.

I’ve also updated the way individual house prices are listed so that all sales for a particular property are grouped together (example)

Saturday, May 02, 2015

UK house price data March 2015

I’ve uploaded the latest Land Registry house price data for March 2015 to my site. Prices continue to chug along at 4-5%, as they have been doing for the last 12 months. The CSV download data now includes a yearly summary and I’ve included charts for this annual data at the postcode district level (monthly data is too volatile to show anything meaningful in such small areas)

The trouble with counties

I include some county information with my UK postcode download data, but I get quite a few questions regarding it. The most common question is why don’t all the postcodes have an associated county. The answer is that county information is only shown for postcodes that are located in an administrative county council. So LA1 postcodes are listed as part of Lancashire but BB1 postcodes aren’t, even though most people would consider Blackburn to be in Lancashire. This page gives you an idea of how this works. Select ‘Counties’ in the dropdown and see all the gaps.

The second question that generally follows is whether I could include county information for each postcode. This is where things get tricky. Have a read of this Wikipedia page on the subject of counties. In short, there are the administrative county councils in use today, there are historical counties whose boundaries have changed many times and there are postal counties that used to be supplied by the Royal Mail.

So if I wanted to add county information for every postcode, my first decision would be which of these to use. The Royal Mail seem pretty keen to get rid of postal counties and the information is not provided with the freely available postcode data, so that’s not an option.

So another option would be to use historical counties. Leaving aside the fact these boundaries changed many times, I’m not sure using them would satisfy most users of my download data anyway. Taking an example from the referenced Wikipedia page, how many people would consider Brixton to be in Surrey?

And the final option would be to fill the county data from the local authority. So Brixton would now be in London, but BB1 would now be in ‘Blackburn with Darwen’. Now in this case it’s obvious that any postcodes in the ‘Blackburn with Darwen’ area should show Lancashire as the county so I could possibly map all these authorities to sensible counties, with some work. But even then, would this provide people with what they want? Kingston would be in London, although some people would expect it to be in Surrey, since Surrey county council is based there (and use Surrey in their address).

So in conclusion, counties are blooming tricky and I suspect no one size would fit all, hence the incomplete set of data on my site. If anyone has an opinion on whether there is a good approach to this, let me know.

Monday, March 30, 2015

Cycling halfway round the world

In July 2012 I started to use Endomondo to track my bike rides. Since then some things have changed in my cycling, I’ve moved from a mountain bike to a road bike, I’ve tried cycling 100 miles in one day (only to be thwarted by the weather)and I’ve started using Strava, but I’ve continued to use Endomondo. And today I reached the milestone of getting halfway round the world.

image

Last year I covered almost 6,000 miles. If I can continue at that level, then in a just over a couple of years I will have completed my virtual trip round the world. If you want to encourage me, then you can sponsor me in this year’s Ride London 100

Saturday, March 28, 2015

Loading Google Maps asynchronously

Google’s PageSpeed keeps telling me I should load scripts asynchronously to improve the performance of my website. Now in an ideal world I’d use something like RequireJS to implement this for all my scripts, but frankly that seems like a bit of a big task and more than likely I’d stuff it up and break large chunks of my web site. So I thought I’d start small and just load up Google Maps asynchronously. Google provide an example of how to do this, but I wanted to encapsulate that in a simple reusable function with a callback function parameter to run after the library had loaded. This is what I came up with in TypeScript.

function loadGoogleMaps(libraries: string, callback: () => void) {

  window["initGM"] = () => {
    callback();
  };

  var script = document.createElement("script");
  script.type = "text/javascript";
  var url: string = "http://maps.google.com/maps/api/js?callback=initGM";
  if (libraries != null && libraries !== "") {
    url += "&libraries=" + libraries;
  }
  script.src = url;
  document.body.appendChild(script);
}

UK house price data February 2015

I’ve uploaded the latest Land Registry data to my site. Prices continue on their seemingly never ending upward march.

Thursday, March 05, 2015

A cross browser XML parser

A post from three years ago detailing how to implement selectSingleNode for XML documents in a cross-browser friendly manner is still getting a good number of hits. Which I guess shows that developers still need to manipulate XML in browsers, even with the increasing popularity of JSON. When we started to rewrite our desktop app on the web, JSON seemed like the obvious choice, but we use XPath in a big way and we wanted our web app to be compatible with our desktop app, so we stuck with XML, which meant having to deal with the different ways XML is supported in different browsers. So we now have a reasonably well featured cross browser XML parser, the source of which you’ll find below.

A couple of things to note, this probably works in IE8 and below but I’ve never tested it since our app needs at least IE9. Also, the code is TypeScript rather than Javascript, since TypeScript is slightly less insane…

// stop TypeScript complaining about stuff we don't have definitions for
interface Window {
  DOMParser;
}
declare var XPathResult;

class XmlWrapper {
  private xmlDoc: any;
  constructor(xml: string) {
    try {
      // try Internet Explorer first. Although later versions have DOMParser, they don't implement evaluate
      this.xmlDoc = new ActiveXObject("Microsoft.XMLDOM");
      this.xmlDoc.async = false;
      this.xmlDoc.loadXML(xml);
      this.xmlDoc.setProperty("SelectionLanguage", "XPath");
    } catch (ex) {
      if (window.DOMParser) {
        var parser = new DOMParser();
        this.xmlDoc = parser.parseFromString(xml, "text/xml");
      } else {
        throw new Error("Can't find an XML parser!");
      }
    }
  }

  public selectSingleElement(xmlNode, elementPath: string): Element {
    return <Element>this.selectSingleNode(xmlNode, elementPath);
  }

  public selectSingleNode(xmlNode, elementPath: string): Node {
    if (xmlNode == null) {
      xmlNode = this.xmlDoc;
    }

    if (this.xmlDoc.evaluate) {
      var doc = xmlNode.ownerDocument;
      if (doc == null) {
        doc = xmlNode;
      }
      var nodes = doc.evaluate(elementPath, xmlNode, null, XPathResult.ANY_TYPE, null);
      var results = nodes.iterateNext();
      return results;
    } else {
      return xmlNode.selectSingleNode(elementPath);
    }
  }

  public selectElements(xmlNode, elementPath: string): Element[] {
    return <Element[]>this.selectNodes(xmlNode, elementPath);
  }

  public selectNodes(xmlNode, elementPath: string): Node[] {
    if (xmlNode == null) {
      xmlNode = this.xmlDoc;
    }

    if (this.xmlDoc.evaluate) {
      var doc = xmlNode.ownerDocument;
      if (doc == null) {
        doc = xmlNode;
      }

      var resultsArray = [];
      var results = doc.evaluate(elementPath, xmlNode, null, XPathResult.ANY_TYPE, null);
      var thisElement = results.iterateNext();
      while (thisElement) {
        resultsArray.push(thisElement);
        thisElement = results.iterateNext();
      }

      return resultsArray;
    } else {
      return xmlNode.selectNodes(elementPath);
    }
  }

  get documentElement(): Element {
    return <Element>(this.xmlDoc.documentElement);
  }

  public xml(): string {
    // for IE
    if (this.xmlDoc.documentElement.xml) {
      return this.xmlDoc.documentElement.xml;
    }

    // Chrome, FireFox
    if (this.xmlDoc.documentElement.outerHTML) {
      return this.xmlDoc.documentElement.outerHTML;
    }

    // Safari
    return (new XMLSerializer()).serializeToString(this.xmlDoc.documentElement);
  }

  public static getNodeText(node: Node): string {
    if (node == null) {
      return "";
    }

    var value: string = node.text;
    if (node.textContent) {
      value = node.textContent;
    }

    return value;
  }

  public static setNodeText(node: Node, value: string): void {
    if (node == null) {
      return;
    }

    if (node.text !== undefined) {
      node.text = value;
    }

    if (node.textContent !== undefined) {
      node.textContent = value;
    }
  }
} 

Sunday, March 01, 2015

Google Earth Pro is kind of free

I was excited when Google announced Google Earth Pro was going to become free. A lot of geographical data comes in Shape files, but these aren’t usable in any free applications I know of and can’t really be used on the web without converting to something like KML first. But Google Earth Pro can load up Shape files and convert them to KML/KMZ.

Downloading isn’t a problem but getting hold of a key doesn’t seem to work as far as I can tell. Following the link to grab a free key just keeps redirecting to the download page. So I gave up in the end and grabbed an illicit key instead. Not sure of the legality of that, but if it’s meant to be free and I still need a key and Google won’t give me one, what choice did I have? Anyway a search for “google earth pro serial key or number” gave me a link to a site that produced a key I could use.

Saturday, February 28, 2015

UK house price data January 2015

I have uploaded the latest Land Registry house price data to my site. Prices continue their gradual ascent.

The BBC reports the blindingly obvious that there is a wide gap in regional house prices. This is clearly visible if you compare my place of birth with my current abode. Blackburn, like many areas outside the South East, has seen pretty much static house prices since 2008. Kingston upon Thames, like most other areas in London, took a bit of a breather in 2008 and then continued on its upward trajectory. I guess the interesting question is whether this is a permanent change or if the differences in regional prices will revert back to their historical average at some point. I can’t pretend to know the answer to that, but I do know prices in London are insane…

Wednesday, February 25, 2015

UK Postcode data for February 2015

The ONS have released the latest version of their postcode dataset and I have now uploaded it to my website. A few sanity checks suggest it is OK but let me know if you spot anything strange

Sunday, February 08, 2015

How to beat a Strava PR on every ride

I think getting to, or maintaining, a healthy weight is fairly straightforward, in principle at least. Match your calorie intake with your calorie burning. Riders on the Tour de France eat 9000 calories a day, but don’t put on any weight for the obvious reason that they burn through all those calories. Eating less has never appealed to me, so the exercise side of the equation is the one I try to work on and it generally works OK for me.

But motivation can be a problem. Cycling is my exercise of choice and a winter of cold, wet and windy weather can rather reduce the will to get out on the road. I’ve found a few things to motivate me in the past, signing up with Endomondo (and trying to beat various personal bests), buying a new bike and training for the Ride London 100 to name a few.

Strava is the latest motivator. Every ride gives the opportunity to beat a PR on one segment or another, so there are many more chances to get a little boost from receiving a medal at the end of a ride. But sometimes, nothing. A ride of an hour may lead to no achievement.

But there is a way to almost guarantee a PR on every ride. First, sign up with veloviewer. This provides even more geeky information about every segment you’ve ridden. Next, filter the segment list for ones you’ve only ridden once or twice. Next, check the weather to find out the prevailing wind. Now find some segments where the wind will be behind you. Then plan your ride to include those segments. And finally, ride the route!

And voila, chances are you will beat one of your PRs on that ride. And even better, you’ll probably have ridden some new segments during your ride, which will now be in your list of potential PR segments.

I’ve been using this technique for the last few months and I’ve still got 140 segments within 5 miles of my house that I’ve ridden 2 or less times. Of course I’ll get bored of this at some point, but maybe I’ve already found my next motivation tool (aside from Ride London 2015), increasing my Eddington number

Saturday, January 31, 2015

Monday, December 15, 2014

Grading bike ride climbs

I’ve updated the route elevation page on doogal.co.uk so you can now grade climbs. It uses the same rating as Strava, which is based on the UCI climb categorization. Hopefully it will be useful if you can’t find a matching segment on Strava.

Saturday, December 06, 2014

Non-geographic postcodes added to UK postcode list

I received a complaint that some postcodes were missing from my UK postcode list that were in the original ONS data. I was a little confused by this, since the import is pretty much automated. But looking at my import code I realised I was ignoring postcodes with no location information. I think initially I’d assumed these were postcodes with duff data so ignored them, but for completeness I’ve now added them to the list, adding a few thousand more postcodes. Since they have no associated location data, there isn’t much interesting information available for each of these postcodes but they may be useful for something… An example.

Saturday, November 29, 2014

UK house sales October 2014

I’ve just finished uploading the latest Land Registry UK house sales data to my website. I then realised my annual percentage change calculation was wrong and have fixed that.

Wednesday, November 26, 2014

UK Postcode data November 2014

I’ve uploaded the latest UK postcode data to my site. There seem to be a few changes to the data (some codes for wards and districts have changed and LSOA data is now available for Scotland). I think I’ve found all the anomalies but let me know if you spot anything odd

Sunday, November 23, 2014

More options when calculating route elevations

I’ve added a couple of options to my page for calculating route elevations. You can now select options to avoid highways and toll roads so it should be even more useful!

Saturday, November 15, 2014

Goodbye Endomondo, hello Strava and VeloViewer

Back in the day I used to go for bike rides just for the fun of it. Then I discovered Endomondo and was able to analyse every part of a ride. And being a geek, that motivated me to ride more, try to ride faster and try to ride further. I could try to beat my best time for 10 miles or distance covered in an hour. But the number of awards was kind of limited, so my motivation started to flag. I got some more enthusiasm by training for RideLondon 100. But once I’d completed that (or to be more accurate, the 86 miles that I was allowed to ride), I was looking for something else to keep me moving, especially now the days are getting shorter and the weather is deteriorating.
I’d heard of Strava before but I’d discounted it because there was no app for my Windows Phone (what’s new right?). But then I realised Endomondo has the ability to export workouts which can then be imported into Strava. So I gave it a try. And I was immediately hooked. Rather than compare activity over a specific distance/time, I can compare efforts over the same bit of road, a segment. So for any ride I might pass through 20 or 30 segments and I generally beat a personal best on at least one of them. And that helps keep me going.
And for the ultimate geek experience, there’s VeloViewer. This runs on top of Strava and gives me an even more in depth view of my performance on segments. It tells me I have cycled on 1472 segments and I have a score of 86.4 (I have no idea if this is good or bad but it’s headed in an upwards direction which seems like a good thing)
VeloViewer was free but is now asking for £9.99 a year, with a pretty limited free version. This seems like a reasonable amount to pay for something so compelling. Strava also has a premium service but I haven’t seen anything in there that I need. My guess is that Strava will probably start to require premium membership to use their API fully which means any VeloViewer users will need to pay for their Strava account to use VeloViewer. I’d have thought that would be a no-brainer for Strava, since all VeloViewer users are the kind of people who are prepared to pay for a service and are therefore likely to cough up for Strava as well.
Anyway, that should keep me going during the long winter months. Maybe my next motivator will be my own little app running on top of Strava…

Friday, November 14, 2014

House price data now with annual change

The Land Registry sales data on my site now includes the annual price change. I thought this would smooth out the seasonal variations but the annual change is still quite volatile. Even so, it may be of interest.

Thursday, October 02, 2014

Unable to get property 'setState' of undefined or null reference in ckeditor.js

I’ve been working on the latest and greatest Business Optix product which dynamically creates lots and lots of HTML controls, some of which are CKEditor controls. Whilst testing the app and quickly navigating around it, I started getting an error “Unable to get property 'setState' of undefined or null reference” coming from deep in the bowels of CKEditor. Google astonishingly turned up nothing, but I remembered I’ve had trouble in the past with CKEditor when I didn’t explicitly destroy the editors. The fix was simple, like this

for (name in CKEDITOR.instances) {
  CKEDITOR.instances[name].destroy();
}

Friday, September 26, 2014

Property sales data for August 2014

I’ve uploaded the latest Land Registry data to my website. People comparing the data on my site to news stories like this may be wondering why my data shows a new all time average price, whereas the BBC say prices are yet to reach their previous peak of November 2007. I believe this is due to the Land Registry seasonally adjusting their figures whereas the average values on my site uses a geometric mean of the raw data.

Saturday, August 30, 2014

UK property sales data for July 2014

The latest Land Registry data for property sales is now uploaded to my site. Not much can be read into a single month’s figures, but the average sale price has reached an all time high, which I guess pleases a government seeking re-election soon and probably dismays first time buyers.

Wednesday, August 27, 2014

Tuesday, July 29, 2014

Land Registry June 2014 data

I’ve uploaded the Land Registry property sales data for June 2014 to doogal.co.uk. Enjoy!

Friday, July 25, 2014

Setting HttpResponse.StatusDescription silently failing

We recently received a complaint from one of our customers. We provide some fairly simple reporting functionality, that allows more technical users to write their own SQL queries. The customer was building a report and when there was a problem with his SQL, sometimes he’d get a detailed error message, but other times he’d get nothing.

When we try to execute a query and an exception is thrown we catch the exception and write the error message to HttpResponse.StatusDescription so it can be displayed in the browser. This can fail if the message is longer than 512 characters long, but this is fairly obvious since setting the StatusDescription property will throw an exception. That clearly wasn’t the problem here.

I finally managed to reproduce the problem but was still confused about what was causing the issue. Then I spotted the difference between the working case and the non-working case. The non-working case contained new line characters. Thinking about it, this was fairly obviously going to be a problem. The HTTP response would not be valid, since the status description header would be split over two lines. But it would certainly be preferable if .NET threw an exception in this case, rather than just silently failing to set the status description.

So our code now looks something like this

        context.Response.StatusCode = 500;
        context.Response.TrySkipIisCustomErrors = true;
        string description = ex.Message.Replace('\n', ' ').Replace('\r', ' ');
        if (description.Length > 512)
          description = description.Substring(0, 512);

        context.Response.StatusDescription = description;
        context.Response.ContentType = "text/plain";
        context.Response.Write("An error occurred - " + ex.Message);

Sunday, June 29, 2014

Land Registry May 2014 data

I’ve uploaded the Land Registry data for May 2014 to doogal.co.uk. For all the talk of a house price bubble, the Land Registry data (arguably the most accurate of all the house price indices) doesn’t seem to show much movement at all over the last few months. Of course it’s a different story in London. There was a story about the housing insanity in Hackney a few months ago, and looking at the sales in East London does show prices shooting up over the last year.

Tuesday, May 27, 2014

ONS postcode data for May 2014 uploaded to doogal.co.uk

I’ve just uploaded the latest postcode data from the ONS to my site. There are over 2.5 million postcodes in there, alive and dead. My data checks suggest everything is in order, but let me know if you find a problem.

Monday, May 19, 2014

Downloading Javascript generated data

I have a number of web pages that generate some data in text areas using Javascript. The only way users could download this data was to copy and paste the contents of these text areas, but I wanted to add a download button to simplify the process. The problem is that this simply isn’t possible in Javascript. The only client-side solutions I’ve seen either require Flash or are not supported in all browsers.

So I came up with a slightly hacky and inefficient solution. The basic idea is to post the data to the server and get the server to return it to the client as a download. The HTML looks like this

      <form action="download.php" method="post">
        <div>
          <input type="hidden" name="fileName" value="locations.csv" />
          <input type="submit" value="Download" />
        </div>
        <textarea id="geocodedPostcodes" style="width:100%;" rows="20" name="data"></textarea>
      </form>

All that is needed is a hidden field that tells the server-side script what the download file name should be and a text area with a name of “data”.

The server-side script is pretty simple, it looks like this

<?php
  header('Content-Disposition: attachment; filename="' . $_POST["fileName"] . '"');

  // add data
  print($_POST["data"]);
?>

All it does is get the requested file name and echo back the data.

It’s seems a bit crazy (and a waste of bandwidth) that this seems to be the only way to achieve a seemingly simple task, but that looks to be the case. I’d be happy to be proved wrong.

Sunday, May 11, 2014

Help me go on a bike ride

Last year I saw the various Ride London rides on the telly and rolling through Kingston and fancied doing it myself. Riding round London and Surrey on traffic-free roads is very appealing, compared to the usual stop-start, take your life in your hands experience of cycling round these parts. So the first chance I had, I applied in the ballot for the Ride London-Surrey ballot. And in January I heard I’d missed out on getting a place.

There was one more option. Sign up with a charity and raise some money and get a guaranteed place. So I decided to try and help out Cancer Research. Why Cancer Research? Primarily because cancer affects so many people at all stages of life but also, on a personal level, one of my partner’s best friends lost her life to cancer a couple of years ago, before she reached the age of 40.

I’ve set up a page for donations, added a link from my website and been amazed by the number of people who don’t know me who’ve already donated. If you’ve found this blog or my website useful, or are just feeling generous, then please consider donating some money. I will certainly appreciate it, as will Cancer Research.