Downloading Javascript generated data

I have a number of web pages that generate some data in text areas using Javascript. The only way users could download this data was to copy and paste the contents of these text areas, but I wanted to add a download button to simplify the process. The problem is that this simply isn’t possible in Javascript. The only client-side solutions I’ve seen either require Flash or are not supported in all browsers.

So I came up with a slightly hacky and inefficient solution. The basic idea is to post the data to the server and get the server to return it to the client as a download. The HTML looks like this

      <form action="download.php" method="post">
        <div>
          <input type="hidden" name="fileName" value="locations.csv" />
          <input type="submit" value="Download" />
        </div>
        <textarea id="geocodedPostcodes" style="width:100%;" rows="20" name="data"></textarea>
      </form>

All that is needed is a hidden field that tells the server-side script what the download file name should be and a text area with a name of “data”.

The server-side script is pretty simple, it looks like this

<?php
  header('Content-Disposition: attachment; filename="' . $_POST["fileName"] . '"');

  // add data
  print($_POST["data"]);
?>

All it does is get the requested file name and echo back the data.

It’s seems a bit crazy (and a waste of bandwidth) that this seems to be the only way to achieve a seemingly simple task, but that looks to be the case. I’d be happy to be proved wrong.

Bluffers Guide to responsive design

A while back I spent a bit of time making doogal.co.uk more mobile friendly. I’d put it off for a long time primarily because I thought it would be really hard. But actually it turned out to be not too tricky. So here is my not so comprehensive guide to making your site mobile friendly with responsive design

The first thing to do is decide at what screen size your design will change from the normal design to the mobile design. My decision was that tablets should see the standard design but mobile phones should see the mobile design. So the media query I use for all my mobile CSS is

@media handheld, only screen and (max-width: 600px), only screen and (max-device-width: 600px) 

Better menus - My standard menus were tricky to use on a small screen, but fortunately they were built using styled unordered lists i.e. <ul></ul>, so I was able to use the Mobilemenu JavaScript library. This converts the list into a dropdown <select> which is much more useable on mobile devices.

Hide stuff – display:none is your friend. Most websites have bits on the page that may be useful but aren’t entirely essential. On a big screen we can get away with that, but on mobile devices it’s necessary to concentrate on the essential information. So hide anything that isn’t needed. I have several tables with many columns, but each row links through to more information, so I hid several of the columns since the tables didn’t render very well. The best approach to this appears to be CSS like the following

.postcodeTable  td + td { display: none;} 

No more table layouts – We’ve been told for years not to use tables for layout of our websites. But I’ve certainly been guilty of it, since it’s always easier to build a multi-column layout using a table. But now is when the chickens come home to roost. You may want two columns on the desktop, but on a mobile device, you’ll probably want to have the two columns stacked on top of each other, so those tables will need to be converted to div’s which float on the desktop and don’t on mobile devices.

Server side control – The solutions so far have all been client-side. I think this is generally the easiest way to deal with the issue, but there is certainly a good argument for saying content shouldn’t be getting pushed to the client if the client will never actually display it. If you’re using PHP on the back-end, Mobile Detect Library can be used to tailor your HTML before it leaves the server. One place where you may need to do this is adverts. Google’s T&Cs say you can’t hide adverts, so using display:none for them is probably a bad idea.

Handler "PHP55_via_FastCGI" has a bad module "FastCgiModule" in its module list

Trying to get a PHP app working on Windows Server 2012, I got the following error message

“HTTP Error 500.21 - Internal Server Error

Handler "PHP55_via_FastCGI" has a bad module "FastCgiModule" in its module list”

I’d installed PHP via the IIS Web Installer but had failed to do one more thing, enable CGI in ‘Add Roles and Features’ in Server Manager, under Web server (IIS)/Web Server/Application Development. After enabling that, everything sprang to life

Installing PHP 5.2 on Windows 7 IIS 7.5

I had a bit of fun upgrading my Windows 7 machine to PHP 5.2 the other day. There’s an installer available that goes most of the way but didn’t quite work for me (there’s also an installer available on www.php.net, I’m not sure how they differ).

The first problem was that my installation of IIS didn’t have the FastCGI module installed. This was pretty easy to rectify, in the IIS administrator go to Modules and then click the ‘Configure Native Modules’ link and then click the ‘Register’ button. The FastCGI module will live somewhere like ‘C:\Windows\System32\inetsrv\iisfcgi.dll’. Then I manually added a handler for PHP files, although if you add the module with the correct ‘FastCgiModule’ name, a re-install should correctly add the handler.

After this was set up, I was able to load up PHP pages but they weren’t being rendered properly, because it looked like the PHP code wasn’t being evaluated and was just getting outputted as plain text. This confused me for a while and I thought the installer hadn’t done its job. But after firing up Fiddler, I saw this header in the response from the server – X-Powered-By: PHP/5.2.14, which suggested the file was getting handed to PHP which was doing something with it. So I guessed there must be a problem somewhere in the PHP config file, php.ini. After searching around in there, I discovered this setting - short_open_tag = Off. This means any PHP files that use <? as their opening tag instead of <?php will not get evaluated, and turning that option on fixed my problems. One other thing to note is that an IIS reset seems to be required for any changes to that config file, since when using FastCGI the PHP executable stays resident in memory rather than restarting for every request.

Building a random postcode generator

Some time a go I noticed that the keywords ‘random postcode generator’ were landing quite a few people at my website, even though I didn’t have a random postcode generator. Which got me thinking there must be a need for such a tool, if people were ending up at a site that didn’t have one desperately looking for it. So I built one and it’s now the most visited page on my site.

So that’s my little success story. But then I started wondering why anybody would want a random postcode anyway. I guess some people are just after a fake postcode they can plug into a website registration form, for whatever reason, and that page will do the trick nicely. But I imagine there are other people who need to generate random postcodes programmatically for some reason, so I’m going to explain how my postcode generator works. It may not be the best approach, but it seems to work.

I guess one approach would be to have a complete list of UK postcodes and just select one at random. I didn’t have a complete list of postcodes and you probably won’t either, unless you’ve bought the PAF database from the Royal Mail. So I went for a more long winded approach that requires less initial data to get working. First I got together a table of postcode districts (original list and my more programmer friendly version), which is the first half of a postcode. Potentially you could try to create this part of the postcode randomly but you’d end up creating invalid postcodes a lot of the time. Also, not all postcode districts are of the same format.

So with this table in place, I get a random postcode district and then append a random second half of the postcode (which is always of the form digit-letter-letter). I did this in PHP using the following (this contains some code which is specific to my website but you should be able to modify for your own needs)

  $result = DbQuery("SELECT Postcode FROM PostcodeDistricts ORDER BY rand( ) LIMIT 1");
  $line = mysql_fetch_array($result, MYSQL_ASSOC);
  $num = rand(0,9);
  $firstChar = rand(1, 26);
  $secondChar = rand(1, 26);
  print($line["Postcode"] . " " . $num . chr(64+$firstChar) . chr(64+$secondChar));
  mysql_free_result($result);

The only problem with this is that it will sometimes generate invalid postcodes, so the next step is to check the postcode is valid. I did this using Google’s Local Search API from JavaScript, as so.

  <script type="text/javascript">
    /* <![CDATA[ */
    function CreateRandomPostcode()
    {
      document.getElementById("postcode").innerHTML = "Thinking";
      TryCreateRandomPostcode();
    }
 
    function TryCreateRandomPostcode()
    {
      document.getElementById("postcode").innerHTML += ".";
      var xmlhttp;
      if (window.XMLHttpRequest) {
        xmlhttp = new XMLHttpRequest();     // Firefox, Safari, ...
      }
      else if (window.ActiveXObject)   // ActiveX version
      {
        xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");  // Internet Explorer
      }
 
      xmlhttp.onreadystatechange = function() {
        if(xmlhttp.readyState == 4 && xmlhttp.status == 200) {
          Geocode(xmlhttp.responseText);
        }
      }
      xmlhttp.open("GET", "CreateRandomPostcode.php", true);
      xmlhttp.send();
    }
 
    function Geocode(postcode)
    {
      var localSearch = new GlocalSearch();
      localSearch.setSearchCompleteCallback(null,
        function()
        {
          if (localSearch.results[0])
          {
            var results = localSearch.results[0];
 
            if ((results.lat > 49) && (results.lat < 61) && (results.lng > -12) && (results.lng < 3)) {
              document.getElementById("postcode").innerHTML = "Your random postcode is " + postcode;
              document.getElementById("lat").value = results.lat;
              document.getElementById("long").value = results.lng;
            }
            else {
              TryCreateRandomPostcode();
            }
          }
          else
          {
            TryCreateRandomPostcode();
          }
        });
 
      localSearch.execute(postcode + ", UK");
    }
 
    /* ]]> */
  </script>

Basically it tries to geocode the postcode and if it fails (or it lies outside the UK), assumes the postcode is not valid and tries a different postcode until a valid one is produced. This could all be done server-side by hacking the Local Search API, but this met my simple needs.

XML encoding in PHP

I’ve needed it in JavaScript in the past and now I need it in PHP. Escape all those pesky special XML characters like so

  function XmlEncode($content)
  {
    $trans = array("&" => "&amp;", "<" => "&lt;", ">" => "&gt;", "'" => "&apos;",
      "\"" => "&quot;", "’" => "&apos;");
  	return strtr($content, $trans);
  }

Improving performance with Gzip compression in PHP

I was perusing through some of the data provided by Google Webmaster Tools when I came upon the Site Performance section. this told me that doogal.co.uk took longer to load than 96% of sites on the web, which was a little embarrassing. I knew what was causing at least some of this slowdown, the incomplete list of UK postcodes which has been getting gradually slower as the number of postcodes has got longer and longer. I’d already started to address that by paging the data, but the CSV data couldn’t really be paged, without reducing the value of it. So I had a look at the suggestion provided by Google, which was to use Gzip compression on the page. I’ve not really thought much about GZip compression before, assuming if it was so useful it would be on by default but I thought I’d give it a go anyway. So I fired up Fiddler and tried downloading my big CSV page and it took approximately 9.5 seconds to fully download. I arrived at this figure as an average after several reloads.

Next I added this line to my PHP source file, ob_start("ob_gzhandler");, and measured the difference. It now took 3.7 seconds to download, wow! A lot of gain for very little effort.

At this point, I wondered if I could add Gzip compression to all my pages. It turns out this is straightforward, just add the following to the .htaccess file

php_flag zlib.output_compression on

After adding this, the CSV page now arrived in 2.7 seconds. I’m not sure why this is even faster, but I’m not complaining. Now the rest of the site feels much snappier as well. So what am I missing? Is there a reason GZip compression isn’t on by default? Am I going to get bitten by this at some point in the future?

How to confuse IE with a 302 redirect

It’s very simple, create a PHP pages called test.php, the contents of which are

<?php
  header("Location: test.php");
  exit;
?>

Now point IE8 (probably earlier versions of IE as well I’d guess) at the page and watch as it tries in vain to fetch the page. It eventually times out, after trying to fetch the same page over 1000 times (take a look with Fiddler to see the problem). FireFox and Chrome are much more sensible in this scenario and give up quite soon. As the dumb developer who caused this problem, it took me quite some time to figure out my redirect wasn’t working rather than the website being very ill

Do you really need to use a CAPTCHA?

I am completely lazy so I've put up with the spam coming to me from the Random Pub Finder for a long while now. Eventually I realised the signal to noise ratio was so low that I was actually missing real people submitting pub suggestions so I decided I had to do something. Again due to my laziness my first attempt at fixing the problem was just to block submissions from the common IP addresses used by the bots trying to submit things. This helped but still stuff was getting through.

So I started to think about implementing a CAPTCHA system, but even though there seems to be at least one good PHP implementation out there, laziness again got the better of me. I remembered Coding Horror which has used a sort of CAPTCHA system for several years but the difference is that the image is always the same. And has been as long as I remember. So I must assume that it is reasonably effective, given it's a site that gets a lot of hits, so I thought I'd try out the same system on the Random Pub Finder, since it's dead simple to implement.

I've only just implemented it and it's working 100% so far. If it continues to work, then it suggests all these spammy form submissions are completed automated and there is no human behind them at all. Which then leads to the conclusion that a full-blown CAPTCHA is probably not necessary for most sites.

How to escape text in SQL statements

Two things got me thinking about escaping text in SQL statements recently. I had previously thought it was a simple topic that everybody already knew about but given that these two things occurred I can only assume that not everybody knows about why escaping text in SQL is important and how to do it. First people keep ending up on the FreeFlow website because one of the methods in the class library is called SqlEscape, so obviously people want to know how to do it. The implementation is very very simple.

    /// <summary>
    /// Escapes a string so that single quotes are replaced by two quotes for use in SQL expressions.
    /// </summary>
    /// <param name="sql">The string to escape.</param>
    /// <returns>The escaped string</returns>
    public static string SqlEscape(string sql)
    {
      if (sql == null)
        return null;
      return sql.Replace("'", "''");
    }

Second, I got hold of some free PHP code off the web that failed to do any escaping of the text in SQL expressions. I foolishly deployed this to a website without checking the code beforehand. Fortunately I managed to spot the error before any dodgy hacker managed to take advantage of it. In the PHP world, escaping can be done using the mysql_real_escape_string function.

So that covers the how but why is this important? The first, probably least important, reason is that it means queries that include ' in text strings will work. I guess most people come across this problem when they try to enter a name like "O'Connor" into an application and it fails.

The more important issue, particularly with public facing websites, is SQL injection where a hacker can manage to run pretty much any query they like against your database.

There are other ways to solve this problem. Stored procedures and parameterised queries will also do the trick.

Handling 404 errors in PHP

If you're running a PHP site on your own server, you can find out what 404 errors have occurred by viewing the log files. If, like me, you're running your PHP site on a cheap and cheerful host, you're probably not going to have access to the log files. But you can probably still do something to find out about them.

The first thing to do is ensure your users see a useful error message, rather than the default provided by your host, which will probably be some page advertising their services. This can be achieved via the .htaccess file, if your site is running on Apache. IIS has similar features, available through its admin tool (although I can't believe there are many people using PHP on IIS). Add something like the following to the .htaccess file to tell Apache to show your custom error page.

  ErrorDocument 404 /error.php

The next thing to do is to make sure you get notified if somebody ends up on a non-existent page on your site, so just add the following code to error.php somewhere.

  
  $message = "URL: " . $_SERVER["REQUEST_URI"] . "\n" . "Referrer: " . $_SERVER["HTTP_REFERER"] . 
    "\n" . "Browser: " . $_SERVER["HTTP_USER_AGENT"];
  mail("you@wherever.com", "404 error", $message);

And that's that. Well except the referrer doesn't seem to work for me... But it's already been useful, I've found one spurious bit of CSS trying to load an non-existent image, although I've also had quite a few false positives.

Getting into MySql with PHP

Our host for the Random Pub Finder has been providing MySql integration for a while. I've been holding off using it for a while because I was thinking of converting the RPF to ASP.NET. But I eventually decided that was going to be too much like hard work and would break any links in. OK, I could probably do some URL rewriting to get round that but again it seemed like a lot of bother and PHP does pretty much everything we need.

Anyway, I signed up for the MySql integration and I've been pleasantly surprised with how good the phpMyAdmin web interface is. RPF has been running off several text files for the last five years so I had to make quite a few changes to our code but getting the data in to MySql was pretty straightforward (phpMyAdmin will suck in any kind of delimited text file). Setting up phpMyAdmin on my local machine was pretty straightforward. OK, it's not MS easy, but then this is a free bit of software developed by people in their spare time. So I had to tweak a few text config files to get it working and I couldn't get it to play with PHP 5 but PHP 4 is fine for our needs. And now it's working I'm presuming I won't need to touch it again.

Updating the RPF live site is now much simpler, I just insert a new row in a table and a new pub appears on the site. Previously I had to update one of our text files and email it to myself, run to my other half's laptop, download the email and FTP it to our host via a phone modem, because they won't let me connect to the FTP site via our broadband connection. I guess that's because their business model is based on getting revenue from phone calls... 

So, in conclusion, a thumbs up for PHP, MySql and phpMyAdmin. Perhaps I'll stop being an MS fanboy and become a open-source zealot. Time to grow a beard...