The random address generator is back

My random addresses generator is back. After having my hand slapped by Google Maps, I’ve had to re-implement it using my own database of property sales. Because of that change, the addresses returned are only from England and Wales, but they will now always be actual addresses rather than the occasional A-Road that used to be returned. And you no longer need to provide a Google Maps API key, yay!

I’ve also removed a few other features which I’ll probably start adding back as time permits. Let me know if there’s anything you desperately need.

Converting KML maps from Google Maps to Here Maps

When Google went insane and decided to charge excessive amounts for use of their mapping APIs I looked for an alternative. One of the features I needed was support for KML, since my website uses it quite extensively. Which led me to Here Maps.

After a fair amount of work, I managed to convert most of my pages to use Here Maps, but there’s still a few stuck using Google Maps due to features that are unique to Google Maps. This was OK, since my usage was now mostly under the $200 per month of free credit. But recently one of my Google Maps pages started to get a lot of hits due to a new inbound link and I zoomed past $200 free credit into eye-wateringly expensive territory. So time to convert that page to Here Maps.

Converting maps that load KML from Google Maps to Here Maps is generally straightforward, just learn a different API and redo the JavaScript on your page. But the architecture of KML support on the two platforms is different. Google loads the KML on their servers, generates map tiles and uses those to display the KML file. Here Maps loads the KML file in the browser and add markers, polylines etc to the map directly.

The Google approach has one major advantage, it copes well with large KML files. Since the move to Here Maps, I’ve had to stop loading up KML files that I know have more than a few thousand markers in them. Google’s KML support also means you can load up KML files from external sources, whereas Here Maps will generally fail with external KML (unless CORS has been configured to support it on the other server).

Google KML has a few disadvantages. If the KML changes regularly then you’ll probably suffer from caching issues, since the old map tiles can keep getting returned for some time after the KML changes. Also the rendering isn’t as good as Here Maps, since the KML is rendered as an image at each zoom level rather than as live objects on the map

The thing that had stopped me moving this page over to Here Maps was the inability to display remote KML data. Then it struck me that the fix for that was fairly straightforward. Add a local piece of server code that loads up the KML file from the remote source and returns it to the browser so it’s treated as a local URL. That was easy enough to code up. Then I just needed to cope with a few edge cases, Google Maps copes with KMZ files, but Here Maps doesn’t. And some servers didn’t like requests coming from something that wasn’t a browser. So I eventually came up with this

public void ProcessRequest(HttpContext context) {
 context.Response.ContentType = "application/vnd.google-earth.kml+xml";
 var url = context.Request.QueryString["url"];
 var httpRequest = (HttpWebRequest) WebRequest.Create(url);
 httpRequest.Method = "GET";
 // pretend to be a browser
 httpRequest.UserAgent =
  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";

 using(var httpResponse = (HttpWebResponse) httpRequest.GetResponse()) {
  var responseStream = httpResponse.GetResponseStream();

  if (responseStream != null) {
   var archiveStream = new MemoryStream();
   responseStream.CopyTo(archiveStream);
   archiveStream.Position = 0;

   // see if it's a zip file
   try {
    var archive = new ZipArchive(archiveStream);
    using(var stream = archive.Entries[0].Open())
    using(var archiveReader = new StreamReader(stream)) {
     context.Response.Write(archiveReader.ReadToEnd());
    }
   } catch (Exception) {
    archiveStream.Position = 0;
    var reader = new StreamReader(archiveStream);
    var response = reader.ReadToEnd();
    context.Response.Write(response);
    reader.Close();
   }

   // Close both streams.

   responseStream.Close();
  }
 }
}

Free for all

As mentioned in a previous post, Google have started charging me for some more Google Maps related things but have also given me some free credit to tide me over. In fact, they think my future usage of Google Maps is going to be so high, they have given me an absolutely huge amount of credits. Since I can’t actually do anything with those credits other than spend them on the Google Maps API, I have removed the requirement for a Google Maps API key on the pages that previously required one. This will remain in place until either my credits run out or we’re close to the end of January. So please go crazy. Happy Christmas.

Important note – if the API key field contains your API key, your API key will still be used!

More Google Maps annoyances

The other day I got another email from Google

Hi,

In June 2016, we announced a change to Maps JavaScript API requests. At that time, we gave you temporary free usage based on your consumption to ensure that your applications would continue to function. The services included in this transition period were: Elevation, Directions, Distance Matrix, Geocoding and Places.

We appreciate you as a loyal and long-standing customer. Our goal is to make sure everyone is on a simple, consistent, and scalable plan with Google Maps Platform.Starting on November 29, 2018, we will bill all your usage for Elevation, Directions, Distance Matrix, Geocoding and Places, according to our new pricing plan.

To help you with this transition, we will provide you with two months of credits, which we will automatically apply to your billing account. Please read our FAQs to understand what these credits cover and how to estimate your monthly bill.

Thank you for using Google Maps Platform.

The Google of 2016 was clearly a different company to the Google of 2018, since grandfathering of old customers when a radical change to pricing is introduced is exactly the right thing to do unlike the recent shenanigans. And generally grandfathering is a permanent thing…

So once again I’m going to have to spend some time switching things off, moving other stuff over to one of Google’s many competitors who have more sane pricing and making other things require a Google Maps API key (the route elevation page now does, sorry)

Implementing my own version of the Google Maps Timezone API

I noticed the other day that my usage of the Google Maps Timezone API was failing. I realised this was down to me not passing in an API key with the call. In their attempts to monetise their Maps API, Google now requires the API key and each call is chargeable. So I added the key and it still didn’t work, although with a different error message. Apparently using a key with a HTTP referrer restriction wasn’t allowed.

So I decided to add a server-side handler on my server that called out to the Timezone API using a server key instead. This was fairly straightforward since it just bounced the AJAX request from the browser to the Timezone API URL.

I checked back the next day to see what my API usage looked like. I’d spent $5 in 24 hours. Continuing with that meant with my other Maps API usage I’d hit the $200 per month free limit and would have to start paying Google money again, something I’ve been loath to do since their ridiculous price increases*

I realised at that point that the Timezone API wasn’t actually doing a huge amount behind the scenes. I guessed there would be libraries out there that could do the same thing but without paying for the privilege. Turns out there is. GeoTimeZone will give the time zone ID for a location and TimeZoneConverter will convert that to a Windows TimeZoneInfo that gives me everything else I needed to build my own version of the Timezone API. The code to do that is something like this

     var lat = double.Parse(HttpContext.Current.Request.QueryString["lat"]);
     var lng = double.Parse(HttpContext.Current.Request.QueryString["lng"]);
     var tz = TimeZoneLookup.GetTimeZone(lat, lng).Result;
// get other info
var tzi = TZConvert.GetTimeZoneInfo(tz);
     // write out as JSON
     var jsonObj = new JObject();
     var rawOffset = tzi.BaseUtcOffset.TotalSeconds;
     jsonObj["dstOffset"] = tzi.GetUtcOffset(DateTime.UtcNow).TotalSeconds - rawOffset;
     jsonObj["rawOffset"] = rawOffset;
     jsonObj["timeZoneId"] = tz;
     jsonObj["timeZoneName"] = tzi.StandardName;
     jsonObj["status"] = "OK";
    var json = JsonConvert.SerializeObject(jsonObj);
     HttpContext.Current.Response.Write(json);

The only thing to consider is that time zones change so it’s worth keeping the two packages up to date.

* For the record, I used to pay about $200 a month to Google. Now I pay about the same to here maps and nothing to Google. I’m intrigued to know how their new pricing has worked out for them, I’m assuming most websites would have made the same decision I did and moved somewhere else.

The economics of the new Google Maps API pricing

The insane new pricing for the Google Maps API with a $200 per month credit leads to an odd situation. For anyone using less than $200 a month, Google Maps is probably the best option (although other map providers also generally have some kind of free tier). For anyone going over that limit, you’d have to be insane to use Google Maps since it now costs so much more than other map providers.

So if you’re in a similar situation to me and you have a bunch of pages using Google Maps, the sensible option is to convert them until you get to a stage where your Google costs are approximately $6.67 per day. The Billings Report in the Google APIs console gives a fairly up to date picture of what’s going on (turn off the ‘Include credit in costs’ option). My current costs are about $20 per day so some work still to be done.

I’d much prefer to be improving the site, but since Google have decided to be dicks, I have to rewrite stuff until I’m no longer paying them a penny…

A reply to Google Maps

I received an email from Google Maps with a subject line of ‘Action needed: Contact Google Maps Platform for volume pricing’, as follows. I thought I’d translate it and reply since it came from a no-reply email address.

Hi,

We are following up on our most recent announcement. This is a reminder that Google Maps Platform’s new terms and pricing will go into effect July 16.

Translation – we are massively increasing our prices on July 16

The 2 months of credit we extended to you will automatically apply to your billing account on that date.

Translation – this won’t cover the increase in prices

You are eligible for a significantly discounted price on your monthly bill, based on your usage over the last three months.

Translation – the discounted price is not that discounted and will still be more expensive than our competitors

If you have not yet contacted us, we strongly recommend that you contact us to learn more about our volume pricing and how it can benefit you.

Reply – I did contact you. Your support staff seem kind of stressed. They sent me off to one of your partners who was only able to offer me a deal that would mean all of my advertising revenue went to Google Maps

Thank you for using Google Maps Platform.

It was fun while it lasted but I’m busily moving all my maps to another provider with more sane pricing. I would really love to know the rationale behind this move because it makes zero sense to me.

Reducing Google Maps costs

I’m sure I’m not the only person who will be hit hard by the huge increase in the cost of Google Maps. I’ve been using Google Maps for years on my site and it’s been absolutely brilliant. But my monthly costs will increase from a couple of hundred pounds to several thousand pounds, which will wipe out any money generated through advertising. And since I was given very little notice of the increase in price, I’ve had to move quickly to try to keep my site useful but still economical. So here’s some suggestions if you’re in the same boat.

Use embedded maps – these are still free and if you’re showing a simple map with a marker then they might meet your needs. You lose Street View but as a quick temporary fix, this seems the most straightforward option.

Don’t load things immediately – I’ve been in the habit of loading maps and showing information from other Google Maps APIs as soon as the page loads, because it hardly cost anything. I’m now looking very closely at what is key information and what may only be useful to some users. If it’s only going to be useful to some users, I’m adding a button to display that data.

Turn things off – sometimes I’ve added stuff just because it looked like fun without considering whether it would be useful at all. Just removing it is generally easier than the previous option.

Other APIs – At the time of writing, my Google APIs console doesn’t even tell me the usage of some APIs that will soon cost a lot of money to use (Places and autocomplete for example). the only suggestion here is to keep an eye on that console because hopefully usage data will appear some time before we start getting charged

Move to a different map provider – If the prices of other map providers remains the same, then I’m not sure why anyone would choose Google Maps anymore. Their prices are completely out of whack with the rest of the world. But maybe they are just the first company to decide they need to charge more because actually it costs a lot to provide a mapping API. If that is the case, the other map providers will probably breathe a sigh of relief and up their prices to a similar level. And if that is the case, moving to another company could be a time-consuming and ultimately fruitless endeavour. That said, the way Google has handled this price rise has made me lose confidence in them so I’m looking at alternatives. I’ve converted one map to Here Maps fairly easily (I chose them mainly because they support KML) and will convert more as time permits

Goodbye Google Maps – important notice

A while back I received an email from Google Maps telling me they were introducing new pricing, but the email included the line “Based on your project usage over the last 3 months and our new pricing plan, we estimate that your new cost will be less than $200 a month and will be covered by our $200 monthly free credit” so I was pretty chuffed since I currently spend about £100 a month on Google Maps. I then promptly forgot about it.

But I then received another email saying the changes were being postponed and I thought it was time to look at how much I’d be paying. After running a few numbers, I realised I’ll have to pay several thousand pounds a month to continue to use Google Maps. Since this will wipe out any advertising revenue I make (via Google of course…), it looks like I’ll have to start removing and replacing various parts of the site.

The Google API Console is fairly useless. Although it tells me my API usage, it doesn’t tell me which pages are causing the usage so I don’t have a clear insight into what I need to remove. But two pages that are likely to be affected are the batch geocoding and batch reverse geocoding pages since they certainly generate a lot of requests. Other pages likely to be affected are random addresses, route elevation and driving distance. Other pages may fail to load maps or fail in other ways after July 16th. Apologies in advance

Google Maps Distance Matrix may not be what you’re after

For many years I’ve been using the Google Maps APIs on my website. It’s been fun to use and until recently the licensing has been very unrestrictive. If an API returned a response saying you’d gone over the query limit, just wait for a second or so, try again and generally it would work. So with some use of setTimeout, it was possible to build reasonably scalable apps that cost nothing.

It’s looking like those days are coming to an end. The various APIs are starting to introduce hard limits on their usage. Once you’re over the limit, that’s it until the counter resets at the start of the next day. I first hit this with my use of the Directions API in my Driving Distances page. I can’t say I’m too happy with the way it was introduced, the Google API Console had given no previous indication of my usage of the API but on the same day as they started displaying the usage report, the hard limit was also introduced. Since my site was way over the limit, that page fell over almost immediately.

So I had a Baldrick cunning plan. I’d swap out the Directions API for the Distance Matrix API. This is exactly the kind of application this API was designed for. Unfortunately I failed to read the usage limits correctly and after I uploaded the new code, the page fell over in a heap again after a few hours. It turns out the usage limits apply to the elements passed to the Distance Matrix API, not the number of requests. So a 10 by 10 matrix counts as 100 towards the free 2,500 limit, not 1 as I had assumed. Given that this API provides less information and has fewer options than the Directions API but has the exact same usage limits, this is rather disappointing!

I am trying to figure out the best way forward now. I could start to pay for extra requests, but since a 100 by 100 matrix would cost $5, the costs could mount up quickly. I can put a maximum daily cost on the account so I don’t have to pay enormous amounts if someone overuses the page, but this could lead to the page becoming unavailable again.

I suspect the outcome will be me removing the page, or at least no longer linking to it from the rest of the site. I make a tidy sum from Google AdSense advertising on the site, but I fear this may just be the start, as more and more APIs start to introduce a hard limit and I don’t particularly want to pay that money back to Google every month to pay for their APIs. It was fun whilst it lasted I guess!

Loading Google Maps asynchronously

Google’s PageSpeed keeps telling me I should load scripts asynchronously to improve the performance of my website. Now in an ideal world I’d use something like RequireJS to implement this for all my scripts, but frankly that seems like a bit of a big task and more than likely I’d stuff it up and break large chunks of my web site. So I thought I’d start small and just load up Google Maps asynchronously. Google provide an example of how to do this, but I wanted to encapsulate that in a simple reusable function with a callback function parameter to run after the library had loaded. This is what I came up with in TypeScript.

function loadGoogleMaps(libraries: string, callback: () => void) {

  window["initGM"] = () => {
    callback();
  };

  var script = document.createElement("script");
  script.type = "text/javascript";
  var url: string = "http://maps.google.com/maps/api/js?callback=initGM";
  if (libraries != null && libraries !== "") {
    url += "&libraries=" + libraries;
  }
  script.src = url;
  document.body.appendChild(script);
}

Google Earth Pro is kind of free

I was excited when Google announced Google Earth Pro was going to become free. A lot of geographical data comes in Shape files, but these aren’t usable in any free applications I know of and can’t really be used on the web without converting to something like KML first. But Google Earth Pro can load up Shape files and convert them to KML/KMZ.

Downloading isn’t a problem but getting hold of a key doesn’t seem to work as far as I can tell. Following the link to grab a free key just keeps redirecting to the download page. So I gave up in the end and grabbed an illicit key instead. Not sure of the legality of that, but if it’s meant to be free and I still need a key and Google won’t give me one, what choice did I have? Anyway a search for “google earth pro serial key or number” gave me a link to a site that produced a key I could use.

More options when calculating route elevations

I’ve added a couple of options to my page for calculating route elevations. You can now select options to avoid highways and toll roads so it should be even more useful!

Google’s weird support for KML

KML is a standard XML format used to describe geographic data. It’s been owned by Google ever since they bought the company who originally developed Google Earth. So you’d think they’d have great support for KML in their products, right? I thought so as well until the other day when I was asked about a problem with some of the KML from my site. So here’s a summary of how different Google products cope with KML.

Google Earth – As you’d expect, Google Earth has full support for KML

Google Maps Javascript API – Using the KmlLayer class, it’s possible to load up KML files from anywhere on the web. There are some limitations on file sizes and not all of the features of the KML file format are supported, but for simple points and polygons it works perfectly well.

Of course, not everyone is a programmer, so using the Javascript API is not really an option for a lot of people. Unfortunately this is where things get messier…

Custom Maps (Maps Engine) – If you try to create a custom map in the new Google Maps, importing from KML is not even an option (the user interface says ‘Google Engine Lite’, so maybe if you pay some money to Google you’ll be able to import KML), which means you have to switch back to…

Custom Maps (Classic) – This claims to support importing KML. My experiments suggests it copes with KML files containing points (up to a fairly small maximum size), but I couldn’t get it working with files containing polygons.

Fusion Tables – Hidden in a corner of the web, is Google Fusion Tables. Basically it’s a spreadsheet with maps and has always seemed like a cool piece of technology but it hasn’t been pushed by Google much. But it seems Fusion’s support for KML is much better than either of the Custom Maps options. Points and polygons worked for me.

What I find weird is that all the web based technologies above are produced by a single company and all do pretty much the same thing, take a KML file and display it on a map, and yet they all clearly use completely different code. Maybe it’s time for some consolidation there?

Bing Maps – If none of the Google options work for you, maybe Bing Maps will do the trick. Points and polygons seem to work, with a couple of restrictions. There’s a fairly low limit on the maximum file size and the Import button only seems available in Internet Explorer.

No doubt there are other options, so let me know if you find any better alternatives.

Traversing JavaScript objects in .NET

I’ve been playing around with hosting a Google Map in a .NET WinForms control, for no good reason other than to see if it can be done. The key part of this integration is calling JavaScript on a web page from .NET. This appears reasonably straightforward, the WebBrowser control has a Document.InvokeScript method that does the trick. If your JavaScript function returns a value, you can pick this up from the return value of InvokeScript.

This works great for simple types, but what if your JavaScript function returns a more complex type? This is where things get a bit more tricky. The returned object has a type of System.__ComObject and initially it looks like this has no useful methods on it. This is the object that is used when calling any COM object, but generally you’ll be able to import a type library to create a .NET friendly wrapper around the raw object. This obviously isn’t the case here.

So my first thought was to find the real underlying type of the __ComObject. This piece of code helped out here. It turns out the actual underlying type was a JScriptTypeInfo but there’s very little information out there about what this type does or how to use it.

But it turns out there’s a much simpler way to access the returned object, cast the returned object as IReflect and use its methods to get properties etc. So the code for my .NET property looks like this

    [Category("Map")]
    public LatLng Centre
    {
      get 
      {
        object centre = webBrowser.Document.InvokeScript("getCentre");
        if (centre == null)
          return new LatLng(0, 0);

        IReflect reflect = centre as IReflect;
        double lat = (double)reflect.InvokeMember("lat", BindingFlags.InvokeMethod, null, 
          centre, null, null, null, null);
        double lng = (double)reflect.InvokeMember("lng", BindingFlags.InvokeMethod, null, 
          centre, null, null, null, null);

        return new LatLng(lat, lng);
      }
      set
      {
        webBrowser.Document.InvokeScript("setCentre", 
          new object[] { value.Latitude, value.Longitude });
      }
    }

Another approach to this would be to have two JavaScript functions, getCentreLat and getCentreLng, which just return simple types, but that could get cumbersome with really complex types.

I’m not sure if it’s possible to pass complex objects into JavaScript but I haven’t needed that yet. Again, for really complex types, passing in each part of them could get messy.

KmlLayer error handling in Google Maps

Two years ago I noted that the KmLayer in Google Maps didn’t provide an event to inform me if there was a problem when loading the KML. Things have moved on and an event has been added to the API at some point. Usage is as follows.

  var kmlLayer = new google.maps.KmlLayer(url, { map: map });
  google.maps.event.addListener(kmlLayer, 'status_changed', function() {
    if (kmlLayer.getStatus() == 'OK')
      $('#status').html('');
    else
      $('#status').html('KML loading problem - ' + kmlLayer.getStatus());
  });

Google Maps in a desktop app

I’ve seen one or two examples of using Google Maps in a WinForms desktop app, but the ones I’ve seen seem to involve loading image tiles from the Google server directly. There’s nothing wrong with that approach but I thought it would be a lot simpler to simply host a local web page using the Google Maps API in a WebBrowser control in an application. Here is a very simple example of this idea.

If you want to extend this example, it’s possible to call scripts in the page via the WebBrowser.Document.InvokeScript method and the application can respond to events in the page via the WebBrowser.ObjectForScripting property.

As an aside, similar ideas can be applied to hosting other Javascript web components, such as an HTML editor like CKEditor.

Google Maps and Friend Connect weirdness in IE8

I received a bug report about my website. The maps that appear on my UK postcodes pages weren’t working in IE8. I hadn’t noticed since they were working fine in IE9 but when I switched to compatibility view in IE9 I started to see the same problem. This helped because I was then able to debug my JavaScript (much as I love IETester, I’d love it even more if I had access to the developer tools for each version of IE). And debugging the script revealed google.maps was null.

At this point I assumed this was a problem with my Google Maps script, so tried loading Google Maps asynchronously then tried specifying an older version of the API. Neither helped. I tried inserting the script in my HTML head but with no luck.

Finally I had a look at the google object and saw the only thing defined in there was friendconnect. Now my suspicions moved away from Google Maps to Google Friend Connect. my Friend Connect stuff appears at the bottom of each page and the script reference for it was also down the bottom of the page. So I thought I’d tried moving the Friend Connect script reference to the html header. And voila, the maps started working again.

So the conclusion? I’m not really sure, although I suspect Friend Connect is removing the google.maps object, although it seems odd that it only happens on IE8.

Styled maps–Showing urban areas

Styled Google Maps but I haven’t found a use for them up until now. But I had a desire to view urban areas in the UK and realised I could do that with styled maps, so here it is.

Loading map...

Debugging KML loading problems in Google Maps

I was attempting to load some KML into Google Maps via the KmlLayer class and not having much luck. Nothing was appearing. Unfortunately the KmlLayer class doesn’t provide an event to indicate an error has occurred. So what is available to debug these problems and what are some of the issues that can cause load errors?

• Fiddler may help, although it didn’t give any indication to me about what was going on. But I have seen some HTTP responses with 400 bad request status codes in the past. Unfortunately they didn’t say why the request was bad.
• Check that the KML is valid using Feed Validator. I’ve found this doesn’t work too well with large KML files in IE, so use some other browser instead.
• When Google Maps does server side rendering of KML it appears to do some caching of the data, so if you change your KML, changes may not appear immediately.
• There are limits to the size of KML file supported by Google Maps, have a look at this page to see if this is your problem. I think this is what caused my issues.
• Try loading the KML in Google Earth to see if it’s an issue with the KML or Google Maps’ handling of it.