GZipping all content served up by ASP.NET

Update – I now realise this post is kind of pointless, there is a module for compression of dynamic content, called unsurprisingly DynamicCompressionModule… But the approach described may be useful for someone somewhere…

I couldn’t find anything that will GZip all the content returned by ASP.NET. There’s a module for compression of static files but nothing for dynamic content. There may be a good reason for this, perhaps the overhead of GZipping content on the fly can kill your server, but since my current project has no static content I thought it would be useful to give it a go. The solution is pretty simple, register the following module in web.config and you’re good to go.

using System;
using System.IO.Compression;
using System.Web;

namespace MyNamespace
{
  public class GzipModule : IHttpModule
  {
    public void Dispose()
    {
      
    }

    public void Init(HttpApplication context)
    {
      context.BeginRequest += new EventHandler(context_BeginRequest);
    }

    void context_BeginRequest(object sender, EventArgs e)
    {
      HttpApplication app = (HttpApplication)sender;
      if ((app.Request.Headers["Accept-Encoding"] != null) &&
            (app.Request.Headers["Accept-Encoding"].Contains("gzip")))
      {
        app.Response.Filter = new GZipStream(app.Response.Filter, CompressionMode.Compress);
        app.Response.AppendHeader("Content-encoding", "gzip");
        app.Response.Cache.VaryByHeaders["Accept-encoding"] = true;
      }
    }
  }
}

Registration is as follows

    <modules>
            <add name="GzipModule" type="MyNamespace.GzipModule" />
    </modules>

Fixing 404 errors when using ASP.NET 4 routing

It took me a while to figure this out. Routing is meant to be baked into ASP.NET 4 but when i tried to set it up, all I got was 404 errors. I did a lot of Googling but couldn’t find anything. It turned out all I was missing was this in web.config

<system.webServer>
    <modules runAllManagedModulesForAllRequests="true"></modules>

Northern Ireland postcode data

The OS Code-Point Open dataset is great, except for a few omissions. It doesn’t include data for Northern Ireland, the Isle of Man or the Channel Islands. It turns out that the Northern Irish postcode data can be found here. Unfortunately that data is in ESRI and MapInfo formats, which I’m not sure how to read. Fortunately Jamie Thompson has converted it to CSV, which is a little easier to deal with.

From that CSV file, it’s quite simple to import the data into SQL Server or MySql using code slightly modified from my Code-Point examples (SQL Server here and MySql here). The only thing to note is that the CSV file uses Irish grid references rather than OS grid references.

Now to figure out where to get hold of the Isle of Man and Channel Islands data…

Don’t believe everything that Reflector tells you

Every .NET developer loves Reflector, since it gives us a chance to see inside assemblies that we don’t have the source for. And I’ve even read bloggers showing off the code that has been reverse engineered by it as evidence of poor coding practices at some organisation or another (“look, these guys use gotos!”). But though Reflector is a brilliant tool, its reverse engineering skills are not perfect. See this fairly innocent looking switch statement from some code I’m working on 

        switch(type)
        {
          case "gateway":
            SetValue(component, "@type", "decision", true);
            string xml = component.InnerXml;
            xml = xml.Replace("gateway", "decision");
            component.InnerXml = xml;
            break;

          case "deliverable":
          case "dataObject":
            SetValue(component, "@type", "document", true);
            break;

          case "annotation":
            SetValue(component, "@type", "note", true);
            break;
        } 

And this is what Reflector shows from the compiled assembly

        if (CS$4$0001 != null)
        {
            if (!(CS$4$0001 == "gateway"))
            {
                if ((CS$4$0001 == "deliverable") || (CS$4$0001 == "dataObject"))
                {
                    goto Label_00B0;
                }
                if (CS$4$0001 == "annotation")
                {
                    goto Label_00C5;
                }
            }
            else
            {
                this.SetValue(component, "@type", "decision", true);
                string xml = component.InnerXml.Replace("gateway", "decision");
                component.InnerXml = xml;
            }
        }
        goto Label_00DA;
    Label_00B0:
        this.SetValue(component, "@type", "document", true);
        goto Label_00DA;
    Label_00C5:
        this.SetValue(component, "@type", "note", true);
    Label_00DA:;

Which I think proves my point…

Better debugging of .NET services

For a long while I’ve been debugging a .NET service using the recommended approach. Whilst this works, it’s kind of painful. the steps are something like this

1. Build the service
2. Realise the service was already running. Stop it from the Services Control Panel applet.
3. Build the service again
4. Start the service from the Services Control Panel applet.
5. Attach to the process from Visual Studio
6. Realise the service has already executed the piece of code I wanted to debug.
7. Goto step 1.

There had to be a better way. And a bit of Googling brought up this approach. But I didn’t really understand how it worked. I guess I’d assumed the error shown in Visual Studio when you try to debug a service was actually coming from Visual Studio, but I now realise the error is coming from .NET. So by having a different piece of code run when in debug mode, the service is treated like any old application.

My solution is slightly different so I can also test the service starting and stopping. My service implementation has StartService and StopService public methods, which are called from OnStart and OnStop. And my Main method looks like this.

    static void Main()
    {
      #if (!DEBUG)
      ServiceBase[] ServicesToRun;
      ServicesToRun = new ServiceBase[] 
            { 
                new MyService() 
            };
      ServiceBase.Run(ServicesToRun);
      #else
      WorkflowService service = new MyService();
      service.StartService();
      service.DoStuff();
      service.StopService();
      #endif               
    }

Onshoring, the new offshoring

Us software developers have had to live with offshoring for years. And I can understand why companies do it. If software isn’t your raison d'être then why not outsource it, and if you’re going to outsource it, why not send it offshore where you can get probably the work done cheaper?

But there are problems with offshoring. There are the time differences and cultural differences that may make it not go as smoothly as hoped. But I’m not one to think there’s any reason why software developed in another country should be any worse than software produced at home. I’ve dealt with many developers, at home and overseas and there are good and bad developers everywhere.

But now things have changed again. The UK government introduced a scheme, the intra company transfer visa, which is intended to bring in people to the UK to fill skills gaps that can’t be filled by the UK population. But big business have found some sweet loopholes in this scheme that means they can pay pretty much minimum wage to their developers whilst claiming back their expenses and not paying NI for them. I can understand why this is attractive to companies, it saves them money after all, and why should they care about the greater good? I doubt Henry Ford would survive very long in the modern world. But I can’t see it’s particularly good for the country as a whole, with lower tax revenue and higher unemployment being the obvious outcome. So what the hell was the government thinking of? I’m quite happy to compete with anybody on a level playing field, but when the playing field has been slanted like this, I’m pretty baffled about what’s going on.

Yesterday we had students smashing up the streets because they think it’s unfair that they are going to be in massive debt after they complete their degrees. In the past, graduates would be the people going into these relatively low paid jobs. How are they going to react when they get out of university to find there are no jobs available for them, due to this wonderful scheme?

Adding a simple website hit counter using the Google Analytics API

Last time, I built a simple app to get the number of visitors to a site using the Google Analytics API. So the obvious next step is to get that data displayed on a web page. The simplest way to do this would be to add some server-side code to an ASP.NET page. But if you’ve run the code in the previous blog entry, you’ll notice it’s pretty slow, generally taking over a second to get the data back from Google. Adding a second to every page load isn’t such a good idea, so this is my slightly more complicated solution, which loads up the hit count via AJAX.

First we need a generic handler, looking something like this

<%@ WebHandler Language="C#" Class="visitorCounter" %>

using System;
using System.Web;
using Google.GData.Analytics;

public class visitorCounter : IHttpHandler 
{
  public void ProcessRequest (HttpContext context) 
  {
    AccountQuery feedQuery = new AccountQuery();
    AnalyticsService service = new AnalyticsService("DoogalAnalytics");
    service.setUserCredentials("email", "password");
    DataQuery pageViewQuery = new DataQuery("https://www.google.com/analytics/feeds/data");
    pageViewQuery.Ids = "ga:103173";
    pageViewQuery.Metrics = "ga:visits";
    pageViewQuery.GAStartDate = DateTime.Now.AddMonths(-1).ToString("yyyy-MM-dd");
    pageViewQuery.GAEndDate = DateTime.Now.ToString("yyyy-MM-dd");
    DataEntry pvEntry = service.Query(pageViewQuery).Entries[0] as DataEntry;

    context.Response.ContentType = "text/plain";
    context.Response.Write(pvEntry.Metrics[0].Value);
  }
 
  public bool IsReusable {
    get {
       return false;
    }
  }
}

Then we just need a bit of jQuery magic to show it on the page.

    <p>
      Visitors this month: <span id="visitorCount"></span>
      <script type="text/javascript">
        $(document).ready(function () {
          $.get('visitorCounter.ashx', function (data) {
            $('#visitorCount').html(data);
          });
        });

      </script>
    </p>

With a little work this could probably be wrapped up as an ASP.NET control, but I’ll leave that as an exercise for the reader.

Simple Google Analytics .NET API usage

So if you want to create your own version of Embedded analytics, the first thing to do is figure out how to use the Google Analytics API. If you’re wanting to use the .NET API, then this little snippet might help you get started. It loops through all the registered sites and shows the number of visits to each site over the last month.

  class Program
  {
    static void Main(string[] args)
    {
      AccountQuery feedQuery = new AccountQuery();
      AnalyticsService service = new AnalyticsService("DoogalAnalytics");
      service.setUserCredentials("email", "password");
      foreach (AccountEntry entry in service.Query(feedQuery).Entries)
      {
        Console.WriteLine(entry.Title.Text);

        DataQuery pageViewQuery = new DataQuery("https://www.google.com/analytics/feeds/data");
        pageViewQuery.Ids = entry.ProfileId.Value;
        pageViewQuery.Metrics = "ga:visits";
        pageViewQuery.GAStartDate = DateTime.Now.AddMonths(-1).ToString("yyyy-MM-dd");
        pageViewQuery.GAEndDate = DateTime.Now.ToString("yyyy-MM-dd");
        foreach (DataEntry pvEntry in service.Query(pageViewQuery).Entries)
        {
          Console.WriteLine(pvEntry.Metrics[0].Value);
        }

      }
      Console.ReadLine();
    }
  }

Embedded Analytics

I’ve been working on a website for somebody and they wanted to show some information about the number of visitors to the site on the site itself. Since we already had Google Analytics installed, the obvious solution was something that hooked into that. I knew there was a Google Analytics API which I could hook into, but being lazy I hoped somebody else had already implemented something for me.

Luckily for me, there is a company called Embedded Analytics that do provide this kind of service. There are various free tools available although it does cost money if you want to use it for multiple sites. But the free service was good enough for my current needs.

Setup was simple, although you do obviously need to let them have access to your Google Analytics data. If you don’t feel comfortable with that then this isn’t the service for you. If you are OK with that, then you can quickly knock together a chart showing the number of visitors or page views in the recent past. Another nice little tool is the ability to get email alerts when somebody has linked to your site.

And now I think I might want to add this to other sites, but it’s going to start costing money to use on more than one site. When my laziness is confronted by my tight fistedness, my tight fistedness always wins out. So now I have to go back to that API and figure out how to do all this and more myself.

Creating a valid file name in C#

This was new to me so I thought I’d share. If you want to generate a file name based on some text, but that text may contain characters that aren’t allowed in file names, what to do? In the past, I’ve fumbled around, replacing invalid characters as bugs have presented themselves. But there is a much more robust way, once I noticed the Path.GetInvalidFileNameChars method. That led to this very simple method.

    private string ReplaceInvalidFileNameChars(string fileName)
    {
      foreach (char invalidChar in Path.GetInvalidFileNameChars())
      {
        fileName = fileName.Replace(invalidChar, '_');
      }

      return fileName;
    }

Styled maps–Showing urban areas

Styled Google Maps but I haven’t found a use for them up until now. But I had a desire to view urban areas in the UK and realised I could do that with styled maps, so here it is.

Loading map...

Uploading files and folders to a specific Google Docs folder using the .NET API

The .NET API for Google Docs is a strange beast. First, it has a DocumentsRequest class and a DocumentsService class. Sometimes you’ll need to use one class, sometimes the other and it’s not clear what the distinction is between the two. And I’d assumed the .NET API supported everything you could do with the raw HTTP API and hence I thought uploading a file or folder to a specific folder wasn’t possible. This led to further confusion when I tried to move a folder and it did move but left the original folder behind and I couldn’t figure out how to delete that folder. And as with most things from Google, the documentation is somewhat lacking.

So I eventually figured out I’d have to do a little work myself and came up with a couple of extension methods, one to create a folder and one to upload a file, but both accepting a parent folder.  When I say I needed to do a little work, what I really mean is I used Reflector to see how the current methods worked and employed some copy and paste…

  public static class GoogleDocsExtensions
  {
    public static Document CreateFolder(this DocumentsRequest docsRequest, string folderName, Document parentFolder)
    {
      Document doc = new Document();
      doc.Type = Google.Documents.Document.DocumentType.Folder;
      doc.Title = folderName;
      if (parentFolder == null)
        return docsRequest.Insert<Document>(new Uri(DocumentsListQuery.documentsBaseUri), doc);
      else
      {
        return docsRequest.Insert<Document>(new Uri(parentFolder.AtomEntry.Content.AbsoluteUri), doc);
      }
    }

    public static void UploadFile(this DocumentsService docsService, string file, Document folder)
    {
      FileInfo info = new FileInfo(file);
      using (FileStream input = info.Open(FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
      {
        Uri uri = new Uri(folder.AtomEntry.Content.AbsoluteUri);

        string str = info.Extension.ToUpper().Substring(1);
        string contentType = (string)DocumentsService.DocumentTypes[str];
        docsService.Insert(uri, input, contentType, info.Name);
      }
    }
  }

Installing PHP 5.2 on Windows 7 IIS 7.5

I had a bit of fun upgrading my Windows 7 machine to PHP 5.2 the other day. There’s an installer available that goes most of the way but didn’t quite work for me (there’s also an installer available on www.php.net, I’m not sure how they differ).

The first problem was that my installation of IIS didn’t have the FastCGI module installed. This was pretty easy to rectify, in the IIS administrator go to Modules and then click the ‘Configure Native Modules’ link and then click the ‘Register’ button. The FastCGI module will live somewhere like ‘C:\Windows\System32\inetsrv\iisfcgi.dll’. Then I manually added a handler for PHP files, although if you add the module with the correct ‘FastCgiModule’ name, a re-install should correctly add the handler.

After this was set up, I was able to load up PHP pages but they weren’t being rendered properly, because it looked like the PHP code wasn’t being evaluated and was just getting outputted as plain text. This confused me for a while and I thought the installer hadn’t done its job. But after firing up Fiddler, I saw this header in the response from the server – X-Powered-By: PHP/5.2.14, which suggested the file was getting handed to PHP which was doing something with it. So I guessed there must be a problem somewhere in the PHP config file, php.ini. After searching around in there, I discovered this setting - short_open_tag = Off. This means any PHP files that use <? as their opening tag instead of <?php will not get evaluated, and turning that option on fixed my problems. One other thing to note is that an IIS reset seems to be required for any changes to that config file, since when using FastCGI the PHP executable stays resident in memory rather than restarting for every request.

WinForms ProgressBar with text

Somebody asked how to use my transparent label on top of a ProgressBar control, since it disappeared when the progress bar changed its state. I had a look at it, but the transparent label wasn’t getting any notification of changes, even when I hooked into the WndProc method. So I wondered if it would be possible to sub-class the ProgressBar control and write some text on top of it that way. And that seems to work OK, as shown below.

Update – I’ve fixed the flickering

Update 2 – It now picks up the text colour from the ForeColor property

Update 3 – Should now work on Windows XP

Update 4 – Now lets you specify the font used for the text

using System.ComponentModel;
using System.Drawing;
using System.Windows.Forms;
using System;

namespace WinFormControls
{
 
  public class TextProgressBar : ProgressBar
  {
    protected override CreateParams CreateParams
    {
      get
      {
        CreateParams result = base.CreateParams;
        if (Environment.OSVersion.Platform == PlatformID.Win32NT
            && Environment.OSVersion.Version.Major >= 6)
        {
          result.ExStyle |= 0x02000000; // WS_EX_COMPOSITED 
        }

        return result;
      }
    }

    protected override void WndProc(ref Message m)
    {
      base.WndProc(ref m);
      if (m.Msg == 0x000F)
      {
        using (Graphics graphics = CreateGraphics())
        using (SolidBrush brush = new SolidBrush(ForeColor))
        {
          SizeF textSize = graphics.MeasureString(Text, Font);
          graphics.DrawString(Text, Font, brush, (Width - textSize.Width) / 2, (Height - textSize.Height) / 2);
        }
      }
    }

    [EditorBrowsable(EditorBrowsableState.Always)]
    [Browsable(true)]
    public override string Text
    {
      get
      {
        return base.Text;
      }
      set
      {
        base.Text = value;
        Refresh();
      }
    }

    [EditorBrowsable(EditorBrowsableState.Always)]
    [Browsable(true)]
    public override Font Font
    {
      get
      {
        return base.Font;
      }
      set
      {
        base.Font = value;
        Refresh();
      }
    }
  }
}

The problems with financial theories

We’re all looking for people to blame for the current mess we are in and apparently there are 38 causes of the current financial crisis. I’m not sure where to get hold of the original report so I’m not sure of what’s on the list but it made me think again about what I learned during my brief time working in the world of high finance a few years ago. I was there for a couple of years, so I'm clearly not hugely knowledgeable but the assumptions made when concocting the financial models used by many investors in the city always seemed a little fishy to me. Here are some of the main assumptions made and the problems I think are associated with them. Almost every financial theory will use one or more of these assumptions.

The market is efficient – All information about an asset is known and priced into the cost of that asset. Peston says this is one of the 38 causes. There are several issues with this theory, my favourite being this paradox – if the market is efficient then there is very little point in trying to play the market, since the market will always be priced accurately so there will be no opportunity to beat it by buying low, selling high. But if there are no players in the market, then it won’t be efficient. Which suggests it can never be perfectly efficient.

Another problem with the efficient market theory can be seen with something like the dot com bubble, where prices bore no relation to their underlying value, how is that sensible? The fact is share prices often bear no relation to the true worth of the company, because people think their value will continue to grow so they will be able to sell them on to a greater fool.

Investors are rational - Given two different assets with the same risk, investors will choose the one with the highest return. Clearly this isn't the case, otherwise bubbles would never happen. Another problem is that even a rational investor may not know the true risk/reward relationship. Shares may fall in value, making their dividend yield look very attractive, but there may be a good reason for this, like the economy is hitting trouble. Finally, some investors are just taking a punt, with no rational thought behind their trading.

Finally, if we decide that not all investors are rational, then we have another reason why the market will not be efficient.

There are no transaction charges – This may be nearly true for large institutional investors but for the man on the street this is very much not true, meaning any kind of investment strategy that involves a lot of trading will incur huge transaction costs, which will eat away at any profits made.

The market is liquid - This assumption basically says if you want to buy or sell, then you can. Whilst this holds true for most shares most of the time, it falls over at the most vital point, when markets are crashing, because everybody is trying to sell and there is nobody willing to buy. It is also not true for quite a few asset classes, as anyone who's tried to sell a house will know.

Normal distribution – A lot of financial theories use the normal distribution for the distribution of volatility/risk. Unfortunately this doesn’t seem to be the case. The black swans seem to occur a lot more frequently than a normal distribution would suggest. For the banks out there, this doesn’t seem to be such a problem, since they take the profits during the good times and get bailed out when those once in a lifetime occurrences strike a little too often.

And it’s the combination of all these flawed assumptions that caused at least some of the problems. They were used to build mathematical models that seemingly abolished risk or certainly incorrectly measured it.

IE9 hardware acceleration not working

I downloaded the IE9 beta a week or so ago. Though it looked pretty good, the performance was less than stellar. When I looked at the options dialog, I saw that ‘Use software rendering instead of GPU rendering’ was checked and also disabled, which made unchecking it quite difficult. Apparently it will be disabled if Internet Explorer decides your graphics card isn’t good enough. I’m not exactly sure what the minimum requirements are, although I have heard rumours that the card must support DirectX 11.

Anyway it seems my old Radeon X600 wasn’t up to the job so I purchased a Radeon HD5450. I have no idea if it’s a good graphics card, but it’s pretty cheap and it does support DirectX 11. After installing that, IE9 was still pretty slow. Until I remembered to uncheck that checkbox…

Structured storage viewer

Before the advent of .NET, Windows development involved a lot of fiddling around with COM interfaces and trying to figure out HRESULT error codes. One prime example of this is the interfaces used to access structure storage. Structured storage is a strange beast itself, providing a way to stitch together composite files, sort of like a seriously complicated version of ZIP files. Of course this was a solution in search of a problem, it turns out ZIP files are pretty much adequate for most situations, which is probably why Office now uses them.

But these technologies have a habit of sticking around (I guess Windows will continue supporting structured storage for the rest of time or until Windows no longer exists, whichever comes sooner). So I was faced with writing some code that had to interact with structured storage. .NET wisely doesn’t have any support for it, so I was faced with dealing with the really horrible COM interfaces. And I was even thinking I’d have to write a little application to let me see inside these composite files, so I’d have a clue what was going on. But fortunately I discovered somebody had already done the job for me, and I finally get to the point of this post. If you need a structured storage viewer, download it from here. http://www.mitec.cz/ssv.html

Converting a Cursor to a Bitmap in .NET

I’ve been playing around with a little tool I’ve been developing and needed to display a Cursor in a PictureBox. There doesn’t seem to be built-in way to convert a Cursor to a Bitmap, which kind of makes sense since it might be an animated cursor so it wouldn’t be sensible to convert it to a Bitmap. I couldn’t find any examples on the web, so thought I’d post this code that does the trick.

          System.Windows.Forms.Cursor cursor = new System.Windows.Forms.Cursor(stream);
          Bitmap bitmap = new Bitmap(cursor.Size.Width, cursor.Size.Height);
          using (Graphics gBmp = Graphics.FromImage(bitmap))
          {
            cursor.Draw(gBmp, new Rectangle(0, 0, cursor.Size.Width, cursor.Size.Height));
            pictureBox.Image = bitmap;
          }

Storing and restoring the clipboard contents

My hacky method to get HTML into Word programmatically had a slight flaw in that it splatted anything the user had placed on the clipboard. I consider it to be bad manners to mess with the clipboard because it’s the user’s clipboard, not my application’s. Many years ago I was forced to use Lotus Notes, which used the clipboard whenever I replied to an email, so I can’t really write code that does something that has annoyed me in the past.

So I had to figure out how to fix it. The below did the trick, although I haven’t tested it with all clipboard formats so it may not work with all of them. And if you’re wondering why this is in VB.NET and the previous example was in C#, I was forced to convert all the code to VB.NET, much to my chagrin.

      ' store current contents of clipboard
      Dim currentData As IDataObject = Clipboard.GetDataObject()
      Dim formats As String() = currentData.GetFormats()
      Dim currentClipboard As Dictionary(Of String, Object) = New Dictionary(Of String, Object)
      For Each Format As String In formats
        currentClipboard.Add(Format, currentData.GetData(Format))
      Next

      ' do stuff...

      ' put back old clipboard contents
      Dim newData As DataObject = New DataObject()
      For Each savedFormat As KeyValuePair(Of String, Object) In currentClipboard
        newData.SetData(savedFormat.Key, savedFormat.Value)
      Next
      Clipboard.SetDataObject(newData)

FxCop 10

When I fired up FxCop 1.36 today, the update check, for probably the first time ever, told me there was a new version available. So I went off to get the update. The odd thing was that I wasn’t pointed towards an installer but a text file. That pointed out that FxCop was now part of the Windows SDK, so installing it is now something of a bigger job than it used to be. In fact, installing the SDK doesn’t install FxCop, it just copies the installer to your machine which then has to be run separately.

But the fact that the version number had increased so much got me excited, thinking there must be lots of improvements… After loading it up and running it against a project, I was a little disappointed. I didn’t spot any new rules and the only difference I noticed was that my custom rules assembly no longer worked.

Updating my rules assembly was pretty simple. First remove the references to the 1.36 assemblies (FxcopSdk.dll and Microsoft.Cci.dll), then update the project to target the .NET Framework 4 (of course this means it has to be built in VS 2010), add references to the version 10 assemblies and rebuild. And it all worked first time. My rules are pretty simple so I don’t know if it will be as easy to upgrade more complex rules assemblies but it all looks pretty good for backwards compatibility.

That said, there doesn’t seem to be any compelling reason to upgrade, unless I’ve missed something.