Showing posts with label C#. Show all posts
Showing posts with label C#. Show all posts

Wednesday, April 13, 2022

Converting .ashx handlers to .NET 5/6

I've been trying to figure out how to convert my site to .NET 5 or 6 for a long while. The pages are written in PHP (because that was the only thing available to me on my web host at the time) but there are many .ashx handlers that were added when I originally decided to move to ASP.NET but never got round to completing the job. But the .NET Framework isn't going to be improved or added to in the future so moving to .NET 5/6 needs to happen at some point. It's not a simple process, the differences are fairly major, but I think I've finally figured out a path to making the move.

My main concern was that I wanted to have a common code base so I can reuse code and continue to work on my current site whilst I set up the updated one, rather than rewriting the whole thing from scratch. A rewrite would certainly be simpler but the switch over would almost certainly be a disaster.

The first step was to work on assemblies used by the site. The Portability Analyzer helped here in figuring out how portable my code was (although it's not generally too helpful in explaining what to do with APIs that are not available in .NET 5). Fortunately it wasn't too difficult to move .NET Coords to .NET Standard 2, meaning it could be used from old and new .NET. I used the .NET Upgrade Assistant to convert to the latest SDK project format. Luckily there were cross platform versions of the third party assemblies I use.

One other thing I did was rip out Entity Framework, which I've never been able to get along with and move from the Oracle MySql ADO.NET connector to MySqlConnector. Like everything from Oracle on Windows, their MySql connector is a complete dumpster fire.

Once that was done, I decided the next step was to move all my .ashx handler code into a new assembly. So every .ashx file now looks something like

<%@ WebHandler Language="C#" Class="DoogalCode.AdministrativeAreasCSV" %>

I configured that assembly to target .NET 4.8 and .NET 5, since there would have to be differences between them. At this point, the .NET 4.8 assembly compiled fine (although with many warnings about using things from System.Web even though that's perfectly fine in .NET 4.8!) and the .NET 5 assembly failed with a gazillion errors. 

The next step was to build the .NET 5 assembly without errors. I decided the simplest way to do that was to build .NET 5 only versions of all the things the compiler was complaining about. This meant things like HttpContext, HttpRequest, HttpResponse and IHttpHandler. The properties and methods didn't do anything except throw an exception, I didn't want the code to run, just compile. I'd figure out the details of  how to implement those methods as I went through each handler and got them working in .NET 5.

Once I'd done that, I needed a .NET web site. Since my front end is written in PHP, I decided to take a look at PeachPie, which claims to be a PHP compiler for .NET. And I am mightily impressed. I created a new PeachPie project, copied my PHP files across and it just worked. I had my website running in .NET 5 in a few minutes. Admittedly I don't do anything very complicated in my PHP, so I can't be sure it's perfect but it's certainly perfect for my needs.

But I still had a bunch of .ashx handlers to get working in .NET 5. I added a method to my website startup class that let me use my dummy IHttpHandler and HttpContext classes to call into my handlers

private static void MapHandler<THandler>(IApplicationBuilder app, string path) where THandler : IHttpHandler, new()

{

  app.Map(path, (app2) =>

  {

    app2.Run(async context =>

    {

      var handler = new THandler();

      await Task.Run(() =>

      {

        handler.ProcessRequest(new DoogalCode.HttpContext(context));

      });

    });

  });

}

Then for each handler I mapped URL paths to the handler via this method, like so

MapHandler<CountiesCSV>(app, "/CountiesCSV");

Now I could call my handlers and start to fix up the methods in my dummy classes. Which is where I am currently, going through each handler fixing issues as I find them, whilst still being able to work on my live website. Once that is done and has gone live, I may try to figure out how to finally move my PHP code to proper .NET

Addendum - This all worked out really well. I was able to retarget all my classic .NET code to .NET 5, whilst continuing to keep the site running. Switching over to the new website was relatively painless, with little downtime. Moving to .NET 6 was straightforward. I was then able to finally start replacing PHP pages with .NET Razor pages, which I'm currently working through. I've also recently learned that Microsoft are developing their own adapters for System.Web.HttpContext etc to make the transition easier, which I wish I'd known sooner!

Friday, January 28, 2022

.NET Coords in .NET Core

.NET Coords is now a .NET Standard 2.0 project, meaning it can be used in .NET Framework 4.8 and .NET Core projects

Saturday, January 01, 2022

C# code to generate a segment FIT file

Someone asked about getting hold of the code to generate FIT files from Strava segments. It's not possible to provide standalone code since it's has dependencies on the rest of my website and a Strava library, but here's what I use which may give people an idea of how to use the rather confusing Garmin FIT API


using System;
using System.Collections.Generic;
using System.IO;
using System.Web;
using Dynastream.Fit;
using Newtonsoft.Json.Linq;
using Strava.Common;
using Strava.Segments;
using Strava.Streams;
public static class FitGenerator { private static void AddLeaderEntry(SegmentPointMesg newRecord, int goal, float currentDistance, float totalDistance, byte goalIndex) { if (goal != 0) { var currentGoalTime = (float)Math.Floor(currentDistance / totalDistance * goal); newRecord.SetLeaderTime(goalIndex, currentGoalTime); } } private static SegmentLeaderboardEntryMesg GetLeaderboard(int goal, byte goalIndex, SegmentLeaderboardType type, string name) { var goalLeaderboard = new SegmentLeaderboardEntryMesg(); if (goal != 0) { goalLeaderboard.SetMessageIndex(goalIndex); goalLeaderboard.SetSegmentTime(goal); goalLeaderboard.SetType(type); if (!string.IsNullOrEmpty(name)) goalLeaderboard.SetName(name); } return goalLeaderboard; } private static int LatOrLngToSemicircle(double latOrLng) { return (int)(latOrLng * (Math.Pow(2, 31) / 180)); } private static bool HasPr(List<SegmentStream> pr, Segment seg) { if (pr != null) return true; if (seg.AthleteSegmentStats != null && seg.AthleteSegmentStats.PrElapsedTime != null) return true; return false; } public static byte[] Generate(Segment seg, int goal, string goalName, int rival, string rivalName, int challenger, string challengerName) { List<SegmentStream> altitude = null; try { StravaBaseHandler.TryStravaRequest((client) => { altitude = client.Streams.GetSegmentStream(seg.Id.ToString(), SegmentStreamType.Altitude | SegmentStreamType.LatLng); }); } catch (Exception ex) { throw new Exception("Failed to get stream for segment " + seg.Id, ex); } // get all the streams we are interested in SegmentStream locationStream = null; SegmentStream altitudeStream = null; SegmentStream distanceStream = null; locationStream = SegmentStream.GetStream(altitude, StreamType.LatLng); altitudeStream = SegmentStream.GetStream(altitude, StreamType.Altitude); distanceStream = SegmentStream.GetStream(altitude, StreamType.Distance); // grab user's best time List<SegmentStream> pr = null; var athlete = HttpContext.Current.Request.Cookies["athlete"]; if (athlete != null) { try { pr = StravaBaseHandler.GetSegmentPr(seg.Id.ToString(), athlete.Value); } catch { // do nothing } } // Generate some FIT messages // Every FIT file MUST contain a 'File ID' message as the first message var fileIdMesg = new FileIdMesg(); var records = new List<SegmentPointMesg>(); fileIdMesg.SetType(Dynastream.Fit.File.Segment); fileIdMesg.SetManufacturer(Manufacturer.Strava); // Types defined in the profile are available fileIdMesg.SetProduct(65534); fileIdMesg.SetTimeCreated(new Dynastream.Fit.DateTime(System.DateTime.Now)); fileIdMesg.SetSerialNumber(1); fileIdMesg.SetNumber(1); var fileCreator = new FileCreatorMesg(); fileCreator.SetHardwareVersion(0); fileCreator.SetSoftwareVersion(0); // segment ID var segmentId = new SegmentIdMesg(); segmentId.SetName(seg.Name); segmentId.SetEnabled(Bool.True); if (seg.ActivityType == "Run") segmentId.SetSport(Sport.Running); else segmentId.SetSport(Sport.Cycling); byte prIndex = 0; byte goalIndex = 1; byte rivalIndex = 2; byte challengerIndex = 3; byte komIndex = 4; byte qomIndex = 5; if (!HasPr(pr, seg)) { goalIndex--; rivalIndex--; challengerIndex--; komIndex--; qomIndex--; } if (goal == 0) { rivalIndex--; challengerIndex--; komIndex--; qomIndex--; } if (rival == 0) { challengerIndex--; komIndex--; qomIndex--; } if (challenger == 0) { komIndex--; qomIndex--; } if (seg.KOM() == 0) { qomIndex--; } segmentId.SetSelectionType(SegmentSelectionType.Starred); segmentId.SetUuid(Guid.NewGuid().ToByteArray()); if (HasPr(pr, seg)) segmentId.SetDefaultRaceLeader(prIndex); if (seg.Map.Polyline == "") throw new Exception("The segment has no map data (which is somewhat unexpected), " + "I can't generate the FIT file"); var points = PolylineDecoder.Decode(seg.Map.Polyline); // figure out SW and NE double swLat = 1000; double swLong = 1000; double neLat = -1000; double neLong = -1000; foreach (var pt in points) { swLat = Math.Min(swLat, pt.Latitude); swLong = Math.Min(swLong, pt.Longitude); neLat = Math.Max(neLat, pt.Latitude); neLong = Math.Max(neLong, pt.Longitude); } // segment info var lap = new SegmentLapMesg(); lap.SetUuid(Guid.NewGuid().ToByteArray()); lap.SetTotalDistance(seg.Distance); lap.SetTotalAscent((ushort)seg.TotalElevationGain); lap.SetSwcLat(LatOrLngToSemicircle(swLat)); lap.SetSwcLong(LatOrLngToSemicircle(swLong)); lap.SetNecLat(LatOrLngToSemicircle(neLat)); lap.SetNecLong(LatOrLngToSemicircle(neLong)); lap.SetMessageIndex(1); lap.SetStartPositionLat(LatOrLngToSemicircle(points[0].Latitude)); lap.SetStartPositionLong(LatOrLngToSemicircle(points[0].Longitude)); lap.SetEndPositionLat(LatOrLngToSemicircle(points[points.Count - 1].Latitude)); lap.SetEndPositionLong(LatOrLngToSemicircle(points[points.Count - 1].Longitude)); // goal entry var goalLeaderboard = GetLeaderboard(goal, goalIndex, SegmentLeaderboardType.Goal, goalName); var rivalLeaderboard = GetLeaderboard(rival, rivalIndex, SegmentLeaderboardType.Rival, rivalName); var challengerLeaderboard = GetLeaderboard(challenger, challengerIndex, SegmentLeaderboardType.Challenger, challengerName); var komLeaderboard = GetLeaderboard(seg.KOM(), komIndex, SegmentLeaderboardType.Kom, ""); var qomLeaderboard = GetLeaderboard(seg.QOM(), qomIndex, SegmentLeaderboardType.Qom, ""); SegmentStream prDistanceStream = null; SegmentStream prTimeStream = null; if (pr != null) { prDistanceStream = SegmentStream.GetStream(pr, StreamType.Distance); prTimeStream = SegmentStream.GetStream(pr, StreamType.Time); } SegmentLeaderboardEntryMesg prLeaderboard = null; if (HasPr(pr, seg)) { prLeaderboard = new SegmentLeaderboardEntryMesg(); prLeaderboard.SetMessageIndex(prIndex); var prTime = seg.AthleteSegmentStats.PrElapsedTime; prLeaderboard.SetSegmentTime(prTime); prLeaderboard.SetType(SegmentLeaderboardType.Pr); } for (var i = 0; i < locationStream.Data.Count; i++) { var newRecord = new SegmentPointMesg(); var latLng = locationStream.Data[i]; if (latLng != null) { var intLat = LatOrLngToSemicircle(Convert.ToDouble(((JArray)(latLng)).First)); newRecord.SetPositionLat(intLat); var intLng = LatOrLngToSemicircle(Convert.ToDouble(((JArray)(latLng)).Last)); newRecord.SetPositionLong(intLng); } if (altitudeStream != null) newRecord.SetAltitude(Convert.ToSingle(altitudeStream.Data[i])); var currentDistance = Convert.ToSingle(distanceStream.Data[i]); newRecord.SetDistance(currentDistance); newRecord.SetMessageIndex((ushort)i); if (HasPr(pr, seg)) { if (pr != null) { // add PR time // find index of the PR item that is at or beyond current distance var pos = prDistanceStream.Data.Count - 1; for (var j = 0; j < prDistanceStream.Data.Count; j++) { if (Convert.ToSingle(prDistanceStream.Data[j]) - Convert.ToSingle(prDistanceStream.Data[0]) >= currentDistance) { pos = j; break; } } // find time at that distance if (pos > 0) { var prStartDistance = Convert.ToSingle(prDistanceStream.Data[pos - 1]) - Convert.ToSingle(prDistanceStream.Data[0]); var prEndDistance = Convert.ToSingle(prDistanceStream.Data[pos]) - Convert.ToSingle(prDistanceStream.Data[0]); var fraction = (currentDistance - prStartDistance) / (prEndDistance - prStartDistance); var prTime = Convert.ToSingle(prTimeStream.Data[pos - 1]) + fraction * (Convert.ToSingle(prTimeStream.Data[pos]) - Convert.ToSingle(prTimeStream.Data[pos - 1])); newRecord.SetLeaderTime(prIndex, (float)Math.Floor(prTime - Convert.ToSingle(prTimeStream.Data[0]))); } else { newRecord.SetLeaderTime(prIndex, 0); } } else { AddLeaderEntry(newRecord, seg.AthleteSegmentStats.PrElapsedTime.Value, currentDistance, seg.Distance, prIndex); } } // add goal time AddLeaderEntry(newRecord, goal, currentDistance, seg.Distance, goalIndex); AddLeaderEntry(newRecord, rival, currentDistance, seg.Distance, rivalIndex); AddLeaderEntry(newRecord, challenger, currentDistance, seg.Distance, challengerIndex); AddLeaderEntry(newRecord, seg.KOM(), currentDistance, seg.Distance, komIndex); AddLeaderEntry(newRecord, seg.QOM(), currentDistance, seg.Distance, qomIndex); records.Add(newRecord); } // Create file encode object var encodeDemo = new Encode(ProtocolVersion.V20); using (var fitDest = new MemoryStream()) { // Write our header encodeDemo.Open(fitDest); // Encode each message, a definition message is automatically generated and output if necessary encodeDemo.Write(fileIdMesg); encodeDemo.Write(fileCreator); encodeDemo.Write(segmentId); if (prLeaderboard != null) encodeDemo.Write(prLeaderboard); if (goal != 0) encodeDemo.Write(goalLeaderboard); if (rival != 0) encodeDemo.Write(rivalLeaderboard); if (challenger != 0) encodeDemo.Write(challengerLeaderboard); if (seg.KOM() != 0) encodeDemo.Write(komLeaderboard); if (seg.QOM() != 0) encodeDemo.Write(qomLeaderboard); encodeDemo.Write(lap); encodeDemo.Write(records); encodeDemo.Close(); fitDest.Flush(); fitDest.Position = 0; return fitDest.ToArray(); } } }

Saturday, April 13, 2019

Converting KML maps from Google Maps to Here Maps

When Google went insane and decided to charge excessive amounts for use of their mapping APIs I looked for an alternative. One of the features I needed was support for KML, since my website uses it quite extensively. Which led me to Here Maps.

After a fair amount of work, I managed to convert most of my pages to use Here Maps, but there’s still a few stuck using Google Maps due to features that are unique to Google Maps. This was OK, since my usage was now mostly under the $200 per month of free credit. But recently one of my Google Maps pages started to get a lot of hits due to a new inbound link and I zoomed past $200 free credit into eye-wateringly expensive territory. So time to convert that page to Here Maps.

Converting maps that load KML from Google Maps to Here Maps is generally straightforward, just learn a different API and redo the JavaScript on your page. But the architecture of KML support on the two platforms is different. Google loads the KML on their servers, generates map tiles and uses those to display the KML file. Here Maps loads the KML file in the browser and add markers, polylines etc to the map directly.

The Google approach has one major advantage, it copes well with large KML files. Since the move to Here Maps, I’ve had to stop loading up KML files that I know have more than a few thousand markers in them. Google’s KML support also means you can load up KML files from external sources, whereas Here Maps will generally fail with external KML (unless CORS has been configured to support it on the other server).

Google KML has a few disadvantages. If the KML changes regularly then you’ll probably suffer from caching issues, since the old map tiles can keep getting returned for some time after the KML changes. Also the rendering isn’t as good as Here Maps, since the KML is rendered as an image at each zoom level rather than as live objects on the map

The thing that had stopped me moving this page over to Here Maps was the inability to display remote KML data. Then it struck me that the fix for that was fairly straightforward. Add a local piece of server code that loads up the KML file from the remote source and returns it to the browser so it’s treated as a local URL. That was easy enough to code up. Then I just needed to cope with a few edge cases, Google Maps copes with KMZ files, but Here Maps doesn’t. And some servers didn’t like requests coming from something that wasn’t a browser. So I eventually came up with this


public void ProcessRequest(HttpContext context) {
 context.Response.ContentType = "application/vnd.google-earth.kml+xml";
 var url = context.Request.QueryString["url"];
 var httpRequest = (HttpWebRequest) WebRequest.Create(url);
 httpRequest.Method = "GET";
 // pretend to be a browser
 httpRequest.UserAgent =
  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";

 using(var httpResponse = (HttpWebResponse) httpRequest.GetResponse()) {
  var responseStream = httpResponse.GetResponseStream();

  if (responseStream != null) {
   var archiveStream = new MemoryStream();
   responseStream.CopyTo(archiveStream);
   archiveStream.Position = 0;

   // see if it's a zip file
   try {
    var archive = new ZipArchive(archiveStream);
    using(var stream = archive.Entries[0].Open())
    using(var archiveReader = new StreamReader(stream)) {
     context.Response.Write(archiveReader.ReadToEnd());
    }
   } catch (Exception) {
    archiveStream.Position = 0;
    var reader = new StreamReader(archiveStream);
    var response = reader.ReadToEnd();
    context.Response.Write(response);
    reader.Close();
   }

   // Close both streams.

   responseStream.Close();
  }
 }
}

Saturday, October 27, 2018

Implementing my own version of the Google Maps Timezone API

I noticed the other day that my usage of the Google Maps Timezone API was failing. I realised this was down to me not passing in an API key with the call. In their attempts to monetise their Maps API, Google now requires the API key and each call is chargeable. So I added the key and it still didn’t work, although with a different error message. Apparently using a key with a HTTP referrer restriction wasn’t allowed.

So I decided to add a server-side handler on my server that called out to the Timezone API using a server key instead. This was fairly straightforward since it just bounced the AJAX request from the browser to the Timezone API URL.

I checked back the next day to see what my API usage looked like. I’d spent $5 in 24 hours. Continuing with that meant with my other Maps API usage I’d hit the $200 per month free limit and would have to start paying Google money again, something I’ve been loath to do since their ridiculous price increases*

I realised at that point that the Timezone API wasn’t actually doing a huge amount behind the scenes. I guessed there would be libraries out there that could do the same thing but without paying for the privilege. Turns out there is. GeoTimeZone will give the time zone ID for a location and TimeZoneConverter will convert that to a Windows TimeZoneInfo that gives me everything else I needed to build my own version of the Timezone API. The code to do that is something like this

     var lat = double.Parse(HttpContext.Current.Request.QueryString["lat"]);
     var lng = double.Parse(HttpContext.Current.Request.QueryString["lng"]);
     var tz = TimeZoneLookup.GetTimeZone(lat, lng).Result;

// get other info
var tzi = TZConvert.GetTimeZoneInfo(tz);

// write out as JSON
     var jsonObj = new JObject();
     var rawOffset = tzi.BaseUtcOffset.TotalSeconds;
     jsonObj["dstOffset"] = tzi.GetUtcOffset(DateTime.UtcNow).TotalSeconds - rawOffset;
     jsonObj["rawOffset"] = rawOffset;
     jsonObj["timeZoneId"] = tz;
     jsonObj["timeZoneName"] = tzi.StandardName;
     jsonObj["status"] = "OK";

    var json = JsonConvert.SerializeObject(jsonObj);
     HttpContext.Current.Response.Write(json);

The only thing to consider is that time zones change so it’s worth keeping the two packages up to date.

* For the record, I used to pay about $200 a month to Google. Now I pay about the same to here maps and nothing to Google. I’m intrigued to know how their new pricing has worked out for them, I’m assuming most websites would have made the same decision I did and moved somewhere else.

Friday, September 22, 2017

Convert a list to a comma string

It’s nothing special, but I’ve found this useful in the past and probably will again so here for my memory bank is a function that converts a list to a comma string


using System.Collections.Generic;
using System.Text;

namespace Utils
{
  public static class ListHelper
  {
    public static string ToCommaString<T>(this IEnumerable<T> list, string separator = ", ")
    {
      var builder = new StringBuilder();

      foreach (var t in list)
      {
        if (builder.Length > 0)
          builder.Append(separator);
        builder.Append(t);
      }

      return builder.ToString();
    }
  }
}

Saturday, May 30, 2015

Retrieving the most popular pages using Google Analytics API again…

I occasionally run a little app I wrote 4 years ago to grab the most popular pages on my website via Google Analytics and update the database so the website can display a top 10 list of those pages. I tried to run it this morning and it fell over in a heap. It seems that Google no longer supports the API I was using. Ho hum, shit happens, software does rust…

So time to drag out my old code. Except I couldn’t find it. So time to look at Google’s latest and greatest API and rebuild it from scratch. Here’s what’s required and a small code sample.

First download the Google API .NET libraries. There seem to be a whole host of Google API libraries in nuget, but for Google Analytics the following should get what you need and all the dependencies.

Install-Package Google.Apis.Analytics.v3

Then you’ll need to create a service account in the Google Developers Console. After creating this service account, create a P12 key for it and save it somewhere on your computer. Then add the service account email address to the Google Analytics account you want to access.

Next fire up Visual Studio and create a console application and add the following code

using Google.Apis.Analytics.v3;
using Google.Apis.Analytics.v3.Data;
using Google.Apis.Auth.OAuth2;
using Google.Apis.Services;
using System;
using System.Security.Cryptography.X509Certificates;

namespace UpdateTop10
{
  class Program
  {
    [STAThread]
    static void Main(string[] args)
    {
      try
      {
        Read();
      }
      catch (Exception ex)
      {
        Console.WriteLine("ERROR: " + ex.Message);
      }
      Console.WriteLine("Press any key to continue...");
      Console.ReadKey();
    }

    private static void Read()
    {
      String serviceAccountEmail = "<the service account email address>";

      var certificate = new X509Certificate2(@"<the location of your P12 key file>",
        "notasecret", X509KeyStorageFlags.Exportable);

      ServiceAccountCredential credential = new ServiceAccountCredential(
         new ServiceAccountCredential.Initializer(serviceAccountEmail)
         {
           Scopes = new[] { AnalyticsService.Scope.AnalyticsReadonly }
         }.FromCertificate(certificate));

      // Create the service.
      var service = new AnalyticsService(new BaseClientService.Initializer()
      {
        HttpClientInitializer = credential,
        ApplicationName = "<your application name in Developer Console>"
      });

      // Run the request.
      DataResource.GaResource.GetRequest req = service.Data.Ga.Get("ga:<the ID of the analytics view found under Admin/View Settings>", DateTime.Now.AddMonths(-1).ToString("yyyy-MM-dd"),
        DateTime.Now.ToString("yyyy-MM-dd"), "ga:visits");
      req.Dimensions = "ga:pagePath";
      req.Sort = "-ga:visits";
      GaData data = req.Execute();

      for (int i=0; i<20; i++)
      {
        Console.WriteLine(data.Rows[i][0] + ": " + data.Rows[i][1]);
      }
    }
  }
}

Fill in the bits between angled brackets with your details and give it a go. The top 20 visited pages from the last month should appear in your output.


Overall it wasn’t too painful, Some of the examples on the web seem to be written for older versions of the API which can cause some confusion and Google have such a huge number of APIs out there, finding the right one can be tricky, but once those hurdles are overcome, it’s reasonably straightforward.

Update - Windows Live Writer currently doesn't work with Blogger accounts, I'm guessing this is the same issue that I was having with my old little app. Hopefully Google told everyone they were turning off ClientLogin support, but it appears Microsoft didn't get the memo... And hopefully it gets fixed soon, because the Blogger editor is effing terrible

Friday, July 25, 2014

Setting HttpResponse.StatusDescription silently failing

We recently received a complaint from one of our customers. We provide some fairly simple reporting functionality, that allows more technical users to write their own SQL queries. The customer was building a report and when there was a problem with his SQL, sometimes he’d get a detailed error message, but other times he’d get nothing.

When we try to execute a query and an exception is thrown we catch the exception and write the error message to HttpResponse.StatusDescription so it can be displayed in the browser. This can fail if the message is longer than 512 characters long, but this is fairly obvious since setting the StatusDescription property will throw an exception. That clearly wasn’t the problem here.

I finally managed to reproduce the problem but was still confused about what was causing the issue. Then I spotted the difference between the working case and the non-working case. The non-working case contained new line characters. Thinking about it, this was fairly obviously going to be a problem. The HTTP response would not be valid, since the status description header would be split over two lines. But it would certainly be preferable if .NET threw an exception in this case, rather than just silently failing to set the status description.

So our code now looks something like this

        context.Response.StatusCode = 500;
        context.Response.TrySkipIisCustomErrors = true;
        string description = ex.Message.Replace('\n', ' ').Replace('\r', ' ');
        if (description.Length > 512)
          description = description.Substring(0, 512);

        context.Response.StatusDescription = description;
        context.Response.ContentType = "text/plain";
        context.Response.Write("An error occurred - " + ex.Message);

Friday, April 18, 2014

The perils of micro-optimisations

A debate has been raging on my website over the use of StringBuilder.AppendFormat in my exception logger code. OK, raging is something of an exaggeration, there have been two comments in two years. But the point made by two people is that rather than

error.AppendLine("Application: " + Application.ProductName);

I should be using

error.AppendFormat("Application: {0}\n", Application.ProductName);

Since this means I wouldn’t be using string concatenation, which is considered bad for performance reasons. My main reason for not doing anything about this is because I’m lazy, but also because the whole point of this code is that it only runs when an exception is thrown, which hopefully is a pretty rare event, so performance is not a major concern.

But then I wondered what the difference in performance is between these two approaches? So I wrote a little test application that looks like this.

    static void Main(string[] args)
    {
      for (int j = 0; j < 10; j++)
      {
        // try using AppendLine
        Console.WriteLine("AppendLine");
        StringBuilder error = new StringBuilder();
        Stopwatch sw = new Stopwatch();
        sw.Start();
        for (int i = 0; i < 1000000; i++)
        {
          error.AppendLine("Application: " + Application.ProductName);
        }
        sw.Stop();
        Console.WriteLine(sw.ElapsedMilliseconds);

        // try using AppendFormat
        Console.WriteLine("AppendFormat");
        error.Clear();

        sw.Restart();
        for (int i = 0; i < 1000000; i++)
        {
          error.AppendFormat("Application: {0}\n", Application.ProductName);
        }
        sw.Stop();
        Console.WriteLine(sw.ElapsedMilliseconds);
      }

      Console.ReadKey();
    }

The results from this app in milliseconds are as follows (reformatted for clarity)

AppendLine 307 315 321 372 394 370 289 298 300 296
AppendFormat 366 360 362 471 353 359 354 365 365 350

So which is quicker? Well it looks like AppendLine might be marginally quicker. But, much more importantly, who the feck cares? We are repeating each operation 1 million times and the time to execute is still less than half a second. Maybe you can pick holes in my test application, but again I would ask who the feck cares? Either approach is really fast.

And this is the main problem with trying to optimise this kind of stuff. We can spend huge amounts of time figuring out if one approach is quicker than another, but a lot of the time is doesn’t matter. Either the code runs quick enough using any sensible approach, or it’s hit so infrequently that even a really poor implementation will work.

Of course we should consider performance whilst writing code, but we should only use particular approaches when we know they are going to produce more performant code. A good example is the StringBuilder class. We can be pretty sure this is going to be better than using string concatenation, otherwise it wouldn’t exist in the first place. That said, if you’re concatenating two strings I really wouldn’t worry about it.

But the key to writing efficient code is to understand what is slow on a computer. Network operations are slow. Disk access is slow. Because of that, anything that requires large amounts of memory (meaning virtual memory i.e. disk access) is slow. Twiddling bits in memory is quick. Fast code is achieved by avoiding the slow stuff and not worrying about the quick stuff.

And once you’ve written your code and found it doesn’t run ask quick as you’d hoped, don’t jump in and replace calls to AppendLine with calls to AppendFormat, profile your application! Every time I profile an application, I’m always amazed at the causes of the performance bottleneck, it’s rarely where I thought it would be.

If you don’t have a profiler, use poor man’s profiling. There are also free profilers available, I quite liked the Eqatec Profiler which seems to be available from various download sites, although it’s no longer available from Eqatec. But whatever you do, don’t get into Cargo Cult Programming

Saturday, October 26, 2013

Getting the icon for the user’s default browser in C#

In my last post I showed how to get hold of the associated icon for a file. Now, say you want to get hold of the icon for the user’s default browser, you may think you can just grab the icon for HTML files and that’ll do the job. Whilst that will work for some users, it won’t work for all of them. The application used to open HTML files does not need to be the same application that is used to open websites. For example changing your default browser to Chrome won’t change the default application for HTML files to Chrome. So here’s an extension to that previous code to get the icon for the default browser. It won’t work for XP since the way default applications are handled has changed but it should work with all later OSes.

    private static string GetDefaultBrowserPath()
    {
      const string userChoice = @"Software\Microsoft\Windows\Shell\Associations\UrlAssociations\http\UserChoice";
      using (RegistryKey userChoiceKey = Registry.CurrentUser.OpenSubKey(userChoice))
      {
        if (userChoiceKey != null)
        {
          object progIdValue = userChoiceKey.GetValue("Progid");
          if (progIdValue != null)
          {
            string progId = progIdValue.ToString();
            const string exeSuffix = ".exe";
            string progIdPath = progId + @"\shell\open\command";
            using (RegistryKey pathKey = Registry.ClassesRoot.OpenSubKey(progIdPath))
            {
              if (pathKey != null)
              {
                string path = pathKey.GetValue(null).ToString().ToLower().Replace("\"", "");
                if (!path.EndsWith(exeSuffix))
                {
                  path = path.Substring(0, path.LastIndexOf(exeSuffix, StringComparison.Ordinal) + exeSuffix.Length);
                }
                return path;
              }
            }
          }
        }
      }

      return null;
    }

    public static Icon GetDefaultBrowserLargeIcon()
    {
      string browserPath = GetDefaultBrowserPath();

      if (!string.IsNullOrEmpty(browserPath))
        return GetLargeIcon(browserPath);

      // last chance (probably XP), just grab the icon for HTML files
      return GetLargeIcon("test.html");
    }

    public static Icon GetDefaultBrowserSmallIcon()
    {
      string browserPath = GetDefaultBrowserPath();

      if (!string.IsNullOrEmpty(browserPath))
        return GetSmallIcon(browserPath);

      // last chance (probably XP), just grab the icon for HTML files
      return GetSmallIcon("test.html");
    }

Getting the associated icon for a file in C#

Sometimes it’s useful to display an icon for a file in an application and probably the best icon to display is whatever the operating system uses. The Windows API provides the SHGetFileInfo function for this purpose, so here’s a little wrapper around it. All the methods take a file name parameter, but the file doesn’t need to actually exist.

using System;
using System.Collections.Generic;
using System.Drawing;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;

namespace TestIcons
{
  public static class ImageUtilities
  {
    [StructLayout(LayoutKind.Sequential)]
    struct SHFILEINFO
    {
      public IntPtr hIcon;
      public IntPtr iIcon;
      public uint dwAttributes;
      [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
      public string szDisplayName;
      [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 80)]
      public string szTypeName;
    };

    static class Win32
    {
      internal const uint SHGFI_ICON = 0x100;
      internal const uint SHGFI_LARGEICON = 0x0; // 'Large icon
      internal const uint SHGFI_SMALLICON = 0x1; // 'Small icon
      internal const uint SHGFI_USEFILEATTRIBUTES = 0x10;
      internal const uint SHGFI_LINKOVERLAY = 0x8000;

      [DllImport("shell32.dll")]
      public static extern IntPtr SHGetFileInfo(string pszPath, uint dwFileAttributes, ref SHFILEINFO psfi, uint cbSizeFileInfo, uint uFlags);

      [DllImport("User32.dll")]
      public static extern int DestroyIcon(IntPtr hIcon);
    }

    public static Icon GetSmallIcon(string fileName)
    {
      return GetIcon(fileName, Win32.SHGFI_SMALLICON);
    }

    public static Icon GetSmallOverlayIcon(string fileName)
    {
      return GetIcon(fileName, Win32.SHGFI_SMALLICON | Win32.SHGFI_LINKOVERLAY);
    }

    public static Icon GetLargeIcon(string fileName)
    {
      return GetIcon(fileName, Win32.SHGFI_LARGEICON);
    }

    private static Icon GetIcon(string fileName, uint flags)
    {
      SHFILEINFO shinfo = new SHFILEINFO();
      IntPtr hImgSmall = Win32.SHGetFileInfo(fileName, 0, ref shinfo, (uint)Marshal.SizeOf(shinfo),
        Win32.SHGFI_ICON | Win32.SHGFI_USEFILEATTRIBUTES | flags);

      if (hImgSmall == IntPtr.Zero)
        return null;

      Icon icon = (Icon)Icon.FromHandle(shinfo.hIcon).Clone();
      Win32.DestroyIcon(shinfo.hIcon);
      return icon;
    }
  }
}

Sunday, April 28, 2013

Traversing JavaScript objects in .NET

I’ve been playing around with hosting a Google Map in a .NET WinForms control, for no good reason other than to see if it can be done. The key part of this integration is calling JavaScript on a web page from .NET. This appears reasonably straightforward, the WebBrowser control has a Document.InvokeScript method that does the trick. If your JavaScript function returns a value, you can pick this up from the return value of InvokeScript.

This works great for simple types, but what if your JavaScript function returns a more complex type? This is where things get a bit more tricky. The returned object has a type of System.__ComObject and initially it looks like this has no useful methods on it. This is the object that is used when calling any COM object, but generally you’ll be able to import a type library to create a .NET friendly wrapper around the raw object. This obviously isn’t the case here.

So my first thought was to find the real underlying type of the __ComObject. This piece of code helped out here. It turns out the actual underlying type was a JScriptTypeInfo but there’s very little information out there about what this type does or how to use it.

But it turns out there’s a much simpler way to access the returned object, cast the returned object as IReflect and use its methods to get properties etc. So the code for my .NET property looks like this

    [Category("Map")]
    public LatLng Centre
    {
      get 
      {
        object centre = webBrowser.Document.InvokeScript("getCentre");
        if (centre == null)
          return new LatLng(0, 0);

        IReflect reflect = centre as IReflect;
        double lat = (double)reflect.InvokeMember("lat", BindingFlags.InvokeMethod, null, 
          centre, null, null, null, null);
        double lng = (double)reflect.InvokeMember("lng", BindingFlags.InvokeMethod, null, 
          centre, null, null, null, null);

        return new LatLng(lat, lng);
      }
      set
      {
        webBrowser.Document.InvokeScript("setCentre", 
          new object[] { value.Latitude, value.Longitude });
      }
    }

Another approach to this would be to have two JavaScript functions, getCentreLat and getCentreLng, which just return simple types, but that could get cumbersome with really complex types.

I’m not sure if it’s possible to pass complex objects into JavaScript but I haven’t needed that yet. Again, for really complex types, passing in each part of them could get messy.

Thursday, August 09, 2012

Executing multiple SQL statements against multiple databases

In my day job, each of the customers running in our hosted environment have their own SQL Server database. As we develop the software that runs on top of these databases, we often need to update the schema of each database. I knocked together a little console application in C# to do the job and here are the interesting parts of it. First, we need to get a list of the available databases, which can be achieved quite easily.

      List<string> DBs = new List<string>();
      // get list of available databases
      SqlCommand command = conn.CreateCommand();
      command.CommandText = "select * from master.sys.databases where DataBase_ID > 4";
      using (SqlDataReader reader = command.ExecuteReader())
      {
        while (reader.Read())
        {
          DBs.Add(reader.GetString(0));
        }
      }
      DBs.Sort();

Next, we need to execute the SQL updates. This isn’t quite as easy as you’d think. If you want to execute multiple SQL statements with GO statements between them, SqlCommand.ExecuteNonQuery won’t handle them. Fortunately SQL Server comes with some assemblies that solve the problem. So you need to add references to Microsoft.SqlServer.ConnectionInfo.dll, Microsoft.SqlServer.Management.Sdk.Sfc.dll and Microsoft.SqlServer.Smo.dll. Once they have been referenced, the following code should be able to handle any SQL you throw at it.

      foreach (string database in DBs)
      {
        Microsoft.SqlServer.Management.Smo.Server server = 
          new Microsoft.SqlServer.Management.Smo.Server(new ServerConnection(conn));
        server.ConnectionContext.ExecuteNonQuery("USE " + database + "\nGO\n" + sql);
      }

Tuesday, March 06, 2012

Finding a UserPrincipal for an email address

I wanted to let users log in to our application using their Windows user name or their email address. In order to achieve this aim, I needed to get hold of a UserPrincipal from the user’s email address so I could then test the password they entered was correct. It took quite a lot of searching to find the relevant code, so I thought I’d post the pertinent part here.

      PrincipalContext context = new PrincipalContext(
        ContextType.Domain, Environment.UserDomainName);
      
      UserPrincipal user = new UserPrincipal(context);
      user.EmailAddress = "test@test.com";

      // create a principal searcher for running a search operation
      PrincipalSearcher pS = new PrincipalSearcher(user);

      // run the query
      PrincipalSearchResult<Principal> results = pS.FindAll();

      foreach (Principal result in results)
      {
        // do something useful...
      }

Saturday, January 21, 2012

Do it yourself inbound link alerts

Embedded Analytics provide a nice service that will email you whenever somebody clicks on a new link to your site. I’ve been signed up for a while and it’s interesting to see who’s linked to my site. But I received an email last week informing me that my site had so many inbound links that I would have to start paying for the service. To be fair the amount they were going to charge me wasn’t a lot, but I couldn’t really justify spending money on something that is essentially just a way to waste a bit of time for me. And I also figured I could probably do the same thing myself through the Google Analytics API, since this is what Embedded Analytics uses.

I’m assuming that Embedded Analytics uses the source for visitors to your site to spot new links. There is a downside to this since it won’t spot links that have been added but have not been clicked on, but generally these won’t be that interesting, since they presumably are links on low traffic sites.

So to implement this requires a few steps. Pull out the data from Google Analytics and store this data somewhere (DB, XML file, whatever). Then next time we pull the data out of Google, check for new URLs in the returned data and send a notification of these new URLs. Embedded Analytics also goes a step further and validates that the links are valid and that the pages containing them are available from the web. I was only really interested in the first part of this solution so have written a piece of code to pull out the URLs using the Google Data API for .NET. The rest of the work is left as an exercise for the reader!

using System;
using Google.GData.Analytics;

namespace GoogleAnalytics
{
  class Program
  {
    static void Main(string[] args)
    {
      AccountQuery feedQuery = new AccountQuery();
      AnalyticsService service = new AnalyticsService("DoogalAnalytics");
      service.setUserCredentials("email", "password");

      DataQuery pageViewQuery = new DataQuery("https://www.google.com/analytics/feeds/data");
      pageViewQuery.Ids = "ga:202885";
      pageViewQuery.Metrics = "ga:visits";
      pageViewQuery.Dimensions = "ga:source,ga:referralPath";
      pageViewQuery.Sort = "ga:source,ga:referralPath";
      pageViewQuery.GAStartDate = DateTime.Now.AddMonths(-1).ToString("yyyy-MM-dd");
      pageViewQuery.GAEndDate = DateTime.Now.ToString("yyyy-MM-dd");

      DataFeed feed = service.Query(pageViewQuery);
      for (int i = 0; i < feed.Entries.Count; i++)
      {
        DataEntry pvEntry = (DataEntry)feed.Entries[i];
        string host = pvEntry.Dimensions[0].Value;
        string path = pvEntry.Dimensions[1].Value;
        Console.WriteLine("http://" + host + path);
      }

      Console.ReadLine();
    }
  }
}

Tuesday, October 11, 2011

Google Maps in a desktop app

I’ve seen one or two examples of using Google Maps in a WinForms desktop app, but the ones I’ve seen seem to involve loading image tiles from the Google server directly. There’s nothing wrong with that approach but I thought it would be a lot simpler to simply host a local web page using the Google Maps API in a WebBrowser control in an application. Here is a very simple example of this idea.

If you want to extend this example, it’s possible to call scripts in the page via the WebBrowser.Document.InvokeScript method and the application can respond to events in the page via the WebBrowser.ObjectForScripting property.

As an aside, similar ideas can be applied to hosting other Javascript web components, such as an HTML editor like CKEditor.

Saturday, June 04, 2011

Retrieving the most popular pages using Google Analytics API

For a long time I’ve shown the most popular pages on the home page of my website. I did this by logging every page that was viewed to the MySql database on the back end. This kind of worked but had a few problems. First, it wasn’t very clever since it couldn’t tell the difference between a real visitor and a search engine bot. Second, since I’ve started to get quite a few visitors (no, really), it was writing a large amount of data to the database.

So I thought there must be a better solution. Figuring that all the information I needed was already being collected by Google Analytics, I thought I could grab this data and dump it into a much smaller with just the page URL and the number of visits (rather than adding a row for every visit). So I coded up a solution using the .NET wrapper around the Google Analytics API. And this is what it looks like (with the database access code removed for clarity). You’ll need to provide your own email address, password and Google Analytics account table ID to get this to work, for obvious reasons.

using System;
using Google.GData.Analytics;

namespace GoogleAnalytics
{
  class Program
  {
    static void Main(string[] args)
    {
      AccountQuery feedQuery = new AccountQuery();
      AnalyticsService service = new AnalyticsService("DoogalAnalytics");
      service.setUserCredentials("email address", "password");

      DataQuery pageViewQuery = new DataQuery("https://www.google.com/analytics/feeds/data");
      pageViewQuery.Ids = "Google Analytics account table ID";
      pageViewQuery.Metrics = "ga:visits";
      pageViewQuery.Dimensions = "ga:pagePath";
      pageViewQuery.Sort = "-ga:visits";
      pageViewQuery.GAStartDate = DateTime.Now.AddMonths(-1).ToString("yyyy-MM-dd");
      pageViewQuery.GAEndDate = DateTime.Now.ToString("yyyy-MM-dd");

      DataFeed feed = service.Query(pageViewQuery);
      for (int i = 0; i < 20; i++)
      {
        DataEntry pvEntry = (DataEntry)feed.Entries[i];
        string page = pvEntry.Dimensions[0].Value.Substring(1);
        string visits = pvEntry.Metrics[0].Value;

        Console.WriteLine(page + ": " + visits);
      }

      Console.ReadLine();
    }
  }
}

Saturday, March 12, 2011

Updating the Code-Point postcode datataset in MySql

Some time ago I imported the Ordnance Survey Code-Point postcode dataset into MySql. It looks like there’s a new version of that dataset available which includes new postcodes so I wanted to update my database. I guess I could just empty the table and re-import the data, but since it takes some time import and the data is live on the web, this wasn’t the ideal solution. Fortunately, MySql has a useful IGNORE keyword which will ignore failed inserts so any old postcodes will be ignored (since the postcode is used as the primary key on the table) whilst new ones are inserted. Of course, this assumes that the latitude and longitude of old postcodes doesn’t change, which I’m hoping is a reasonable assumption. So my new code looks like this.

using System;
using System.IO;
using DotNetCoords;
using LumenWorks.Framework.IO.Csv;
using MySql.Data.MySqlClient;

namespace ImportCodepoint
{
  class Program
  {
    static void Main(string[] args)
    {
      string[] files = Directory.GetFiles(@"C:\Users\Doogal\Downloads\codepo_gb\Code-Point Open\Data");
      foreach (string file in files)
      {
        ReadFile(file);
      }

    }

    private static void ReadFile(string file)
    {
      using (StreamReader reader = new StreamReader(file))
      {
        CsvReader csvReader = new CsvReader(reader, false);
        using (MySqlConnection conn = new MySqlConnection(
          "server=server;uid=username;pwd=password;database=database;"))
        {
          conn.Open();
          foreach (string[] data in csvReader)
          {
            string postcode = data[0];
            // some postcodes have spaces, some don't
            if (postcode.IndexOf(' ') < 0)
              postcode = data[0].Substring(0, data[0].Length - 3) + " " + data[0].Substring(data[0].Length - 3);
            // some have two spaces...
            postcode = postcode.Replace("  ", " ");
            
            double easting = double.Parse(data[10]);
            double northing = double.Parse(data[11]);

            // there are some postcodes with no location
            if ((easting != 0) && (northing != 0))
            {
              // convert easting/northing to lat/long
              OSRef osRef = new OSRef(easting, northing);
              LatLng latLng = osRef.ToLatLng();
              latLng.ToWGS84();

              using (MySqlCommand command = conn.CreateCommand())
              {
                Console.WriteLine(postcode);
                command.CommandTimeout = 60;
                command.CommandText = string.Format(
                  "INSERT IGNORE INTO Postcodes (Postcode, Latitude, Longitude) " +
                  "VALUES ('{0}', {1}, {2})",
                  postcode, latLng.Latitude, latLng.Longitude);
                int count = command.ExecuteNonQuery();
                if (count > 0)
                  Console.WriteLine("Added");
              }
            }
          }
        }
      }
    }
  }
}

Tuesday, February 22, 2011

GZipping all content served up by ASP.NET

Update – I now realise this post is kind of pointless, there is a module for compression of dynamic content, called unsurprisingly DynamicCompressionModule… But the approach described may be useful for someone somewhere…

I couldn’t find anything that will GZip all the content returned by ASP.NET. There’s a module for compression of static files but nothing for dynamic content. There may be a good reason for this, perhaps the overhead of GZipping content on the fly can kill your server, but since my current project has no static content I thought it would be useful to give it a go. The solution is pretty simple, register the following module in web.config and you’re good to go.

using System;
using System.IO.Compression;
using System.Web;

namespace MyNamespace
{
  public class GzipModule : IHttpModule
  {
    public void Dispose()
    {
      
    }

    public void Init(HttpApplication context)
    {
      context.BeginRequest += new EventHandler(context_BeginRequest);
    }

    void context_BeginRequest(object sender, EventArgs e)
    {
      HttpApplication app = (HttpApplication)sender;
      if ((app.Request.Headers["Accept-Encoding"] != null) &&
            (app.Request.Headers["Accept-Encoding"].Contains("gzip")))
      {
        app.Response.Filter = new GZipStream(app.Response.Filter, CompressionMode.Compress);
        app.Response.AppendHeader("Content-encoding", "gzip");
        app.Response.Cache.VaryByHeaders["Accept-encoding"] = true;
      }
    }
  }
}
Registration is as follows
    <modules>
            <add name="GzipModule" type="MyNamespace.GzipModule" />
    </modules>

Monday, January 31, 2011

Don’t believe everything that Reflector tells you

Every .NET developer loves Reflector, since it gives us a chance to see inside assemblies that we don’t have the source for. And I’ve even read bloggers showing off the code that has been reverse engineered by it as evidence of poor coding practices at some organisation or another (“look, these guys use gotos!”). But though Reflector is a brilliant tool, its reverse engineering skills are not perfect. See this fairly innocent looking switch statement from some code I’m working on 

        switch(type)
        {
          case "gateway":
            SetValue(component, "@type", "decision", true);
            string xml = component.InnerXml;
            xml = xml.Replace("gateway", "decision");
            component.InnerXml = xml;
            break;

          case "deliverable":
          case "dataObject":
            SetValue(component, "@type", "document", true);
            break;

          case "annotation":
            SetValue(component, "@type", "note", true);
            break;
        } 

And this is what Reflector shows from the compiled assembly

        if (CS$4$0001 != null)
        {
            if (!(CS$4$0001 == "gateway"))
            {
                if ((CS$4$0001 == "deliverable") || (CS$4$0001 == "dataObject"))
                {
                    goto Label_00B0;
                }
                if (CS$4$0001 == "annotation")
                {
                    goto Label_00C5;
                }
            }
            else
            {
                this.SetValue(component, "@type", "decision", true);
                string xml = component.InnerXml.Replace("gateway", "decision");
                component.InnerXml = xml;
            }
        }
        goto Label_00DA;
    Label_00B0:
        this.SetValue(component, "@type", "document", true);
        goto Label_00DA;
    Label_00C5:
        this.SetValue(component, "@type", "note", true);
    Label_00DA:;

Which I think proves my point…