Free XML sitemap generator with unlimited pages

I’ve been happily using this free XML sitemap generator for a while, but as my sites have got bigger I keep hitting the 500 page limit. Since I’m too tight to actually pay for this kind of service I’ve been searching around for a while for another free alternative. I was even thinking of writing my own, but before I was forced down that road, I found this alternative generator. I’m not too keen on Java applets generally, but this one seems to work very well. If it fails to download any pages, you can go back and retry downloading the failed pages. It also gives details of how long pages take to download, so it can also be a useful performance checker of a website.

Even so, I’m a little ambivalent towards sitemaps, especially if they are generated using one of these tools that just crawl your website to find all the pages. What information does this provide to search engines that they can’t find for themselves by crawling the site themselves? And If I do want to provide any more useful information, I have to edit the thing by hand, which I really can’t be bothered doing.

Improving performance with Gzip compression in PHP

I was perusing through some of the data provided by Google Webmaster Tools when I came upon the Site Performance section. this told me that doogal.co.uk took longer to load than 96% of sites on the web, which was a little embarrassing. I knew what was causing at least some of this slowdown, the incomplete list of UK postcodes which has been getting gradually slower as the number of postcodes has got longer and longer. I’d already started to address that by paging the data, but the CSV data couldn’t really be paged, without reducing the value of it. So I had a look at the suggestion provided by Google, which was to use Gzip compression on the page. I’ve not really thought much about GZip compression before, assuming if it was so useful it would be on by default but I thought I’d give it a go anyway. So I fired up Fiddler and tried downloading my big CSV page and it took approximately 9.5 seconds to fully download. I arrived at this figure as an average after several reloads.

Next I added this line to my PHP source file, ob_start("ob_gzhandler");, and measured the difference. It now took 3.7 seconds to download, wow! A lot of gain for very little effort.

At this point, I wondered if I could add Gzip compression to all my pages. It turns out this is straightforward, just add the following to the .htaccess file

php_flag zlib.output_compression on

After adding this, the CSV page now arrived in 2.7 seconds. I’m not sure why this is even faster, but I’m not complaining. Now the rest of the site feels much snappier as well. So what am I missing? Is there a reason GZip compression isn’t on by default? Am I going to get bitten by this at some point in the future?

Paste from Visual Studio naughtiness

There’s a nice add-in for Windows Live Writer called Paste from Visual Studio that lets you copy and paste syntax highlighted code from Visual Studio into Live Writer. I’ve used it for a while and had no problems with it. But the other day I looked at the source of the generated HTML and was a little disappointed to find a link to the author’s website with no link text, meaning it never shows up on the page.

I guess it’s OK wanting to get some links to his site, but to do it in such a way as to be invisible to the end user is a little underhand. I’m not sure what a search engine would make of a link like that. Very possibly it will be considered as some kind of black hat SEO and my pages will be downgraded as a result.

So I guess I’ll have to re-implement the plug-in myself. The guy does kindly provide the source code on his website so it shouldn’t be too difficult to get my own version built. Or I guess I can go back to using the excellent C# code format site, which does the right thing by just adding a comment to the generated HTML that includes the URL for the site.

IIS Search Engine Optimization Toolkit

I love Google Webmaster Tools since it tells me things I’m doing wrong on the websites I maintain (in fact I love any tools that tell me what I’m doing wrong, FxCop, the HTML editor in Visual Studio, compilers… I’m sure a psychiatrist would draw some scary conclusions from this admission). But the problem with the Google webmaster Tools is that they aren’t very responsive. If I fix an issue, I have to wait for the Google bot to crawl that page again before I know it’s been fixed.

So I was pleased to be directed to the Search Engine Optimization Toolkit by ScottGu’s post. It’s only in beta but it’s already a powerful tool. It will tell you about lots of potential problems on your site, such as missing alt tags on images, missing description meta tags, broken links, no h1 tag etc… I’ve spent the day trying it out with the Process Mapping site and have cleaned up a lot of potential problems, that no other tools had told me about. Only time will tell if this improves our ranking a lot, but I’m guessing even in the worst case, it won’t cause our ranking to drop, since all the suggestions seem perfectly sensible.

It’s simple to install, requires nothing special on the website to get it working and provides instant feedback, so I’m sold on it. The only possible downside is that it will only install on IIS 7 I guess (although that doesn’t mean your website needs to be running IIS 7, just the machine running the toolkit). 

The search for some page rank

I played around with bing today to see if it’s any good. I was pleased to see that a search for Metastorm brought up our forum and the FreeFlow web page on the first page, so clearly bing is a complete success. Or perhaps not, the image search brought up a photo of their former CEO, who hasn’t been there for many years.

But it did make me go off and check the same search against some other search engines (if you try this yourself, you may well get different results, I’d be interested to know if they are wildly different). Yahoo brings up the Process Mapping website on the first page, Cuil has Jerome’s book on the first page and puts the Process Mapping website on the second page and Ask has the forum on the second page (although for some reason they think I’m Dutch).

Finally I checked the other search engine that you may have heard of. OK, it’s the only search engine that matters. And we are way down the search rankings (page 5 for the forums, page 4 for FreeFlow). But the weird thing is if I search on Google’s UK site, FreeFlow and the forums are on the first two pages. That may make some sense for the FreeFlow site since it’s hosted in the UK, but the forums are hosted in Australia. Next I changed my search term to ‘Meta storm’ and was asked ‘Did you mean: Metastorm’, and although all the results were related to Metastorm, both the FreeFlow website and the forums now appeared on the first two pages. We also rank highly on searches for ‘Metastorm development’ and ‘Metastorm consultancy’ so it seems odd that we are so low down on that one search.

So is it a cock-up, conspiracy, some weirdness in the ranking algorithm or have we been marked down for doing something bad? I don’t think we’ve employed any bad practices in our attempts to improve our ranking and if we had, I would imagine that would impact all our rankings, not just on one search, so I think we can eliminate the last option. But I have no idea about the other three.

Content is still king

One of the phrases that was coined during the early years of the web was 'content is king'. The basic idea was that content drives traffic to your site. This is clearly correct for the simple case - if you have no content at all you'll get no visitors and when you add some content you'll hopefully get more than zero visitors.

But that was in the days of not so clever search engines. These days search engines don't just take notice of the content on your site, they look at all kinds of other things so there's much more emphasis on using search engine optimisation to optimise content to get as many hits as possible. I don't think there's anything wrong with using SEO and fiddling around with keywords, layouts, titles, URLs and reciprocal linking can certainly help (although it's quite possible to get it wrong as well and lose traffic) but it seems to miss the point. The absolute simplest way to increase traffic to your site is to add more content.

To illustrate, lets take a look at some real data. I've used archive.org and data from Google Analytics to produce this graph for the Random Pub Finder.

What does this show? I've taken the number of monthly visitors to the site and divided them by the number of pubs reviews on the website. This doesn't give the complete picture of the amount of content on site but I'm not sure how I can work out the number of unique URLs historically. Note, using page views instead of visitors produces a pretty similar graph.

So does it prove anything? I guess if content was the only thing affecting the number of visitors then we'd expect the trend line to be horizontal so there are clearly other factors at work. We've got more inbound links than we used to have (that big spike is when we got linked from another site). I also think age of content helps search engine rankings, although I don't have any proof for this theory. But given that I've done very little SEO work on the site, it does suggest increasing content increases visitors above and beyond a direct linear increase.

As a comparison, looking at the number of visitors to www.doogal.co.uk, where the amount of content has not increased very much, shows visitor numbers haven't increased as dramatically.

To look at it another way, every new page you add to your site will add X new visitors to your site, where X can be anything from zero to a very large number. Adding a new page will probably increase the number of visitors to your site, however small.

So in conclusion, don't waste your time and money on snake oil SEO 'experts', just add some more content to your site if you want to get more visitors.

How to use Experts Exchange

I don't know about you but a lot of my web searches bring up Experts Exchange articles. When you first look at the page, it looks like you have to register to see the responses to the query. In fact it says "All comments and solutions are available to Premium Service Members only". But look a little further down the page (OK a lot further) and you'll see all the responses without paying a thing.

It may seem surprising that the comments are all visible to anybody who hasn't registered, kind of destroying their whole business model. But I guess they have a problem. I'm guessing most of their traffic comes from search engines so they need to have some decent content to get visitors in. To do that, they need to include all the responses. They could hide the text from the browser whilst making it visible to search engines, but this would almost certainly get them removed from the search engine listings, since this would be considered as black hat SEO. So they have to include the responses visible to everybody and just hope they get enough suckers signed up with the huge blocks of text before you get to the real content.

Great titles don't always lead to lots of readers

Some time ago Nick Bradbury (author of the marvelous FeedDemon which is now free!) said that the best way to increase feed readership is to use great titles. Whilst this is undoubtedly true if your blog has a lot of subscribers, the fact is most of us Z-list bloggers don't actually have many people who subscribe to our blogs. According to Google Sitemaps I currently have two people signed up via Google Reader, which I'm pretty astounded by given my tendency to ramble on incoherently about random subjects.

The fact is 90% of my hits come from Google and Google doesn't care about great titles, it cares about matching up search terms to relevant content. One good way to drive traffic to your site is to write about things that interests searchers and that other people aren't writing about, given that if some A-list blogger does write about it you'll get shoved down the search result list. For me, SonicWall, HTML tick marks and Vista sound problems are the most popular subjects...

Anyway, the things to have in your titles are the words people are likely to use when they are searching for something related to your post. If it's in the title, Google will consider the text more important than if it's just in the body of the post somewhere. So forget great titles, have relevant titles.

The number one internet resource for Metastorm crap

It's always interesting to look through the search terms that have led people here or to the other sites I have something to do with. It's certainly a great work avoidance tactic. So I was pleased to see that the term Metastorm crap had landed somebody here. Then when I did a search for Metastorm crap myself I was even more pleased to see I'm number one for that search term. So there you are, come here for all the Metastorm crap you could ever need...

In fact I've just done a search for Metastorm shit and I'm number one for that as well! I must point out at this point that although the words may have appeared on the same page, they didn't actually appear in the same context... OK, they do now, but I'm not implying anything, OK?

Some useful web resources

I came across a couple of useful websites today which I thought I'd post here for others and for my own information.

First is Domain Tools, which provides the usual Who Is lookup and a whole lot more. Most useful I thought was the SEO text browser that shows how a website looks to a search engine and gives a few pointers to how your pages could be improved for SEO.

That site led me to About Us, that is an interesting attempt at generating a Wiki for websites. I imagine this could become an interesting site to browse and find new sites to visit. If nothing else, adding your site will add another inbound link to your site.

Another useful resource that I didn't find today but follows on from the SEO text browser above is the HTML Validator provided by the W3C. Running your site through this will show any problems with your HTML code. I've no idea if valid HTML will improve your search rankings but I'm fairly certain it won't hinder them.

Google, you are such a tease

You really are. The Random Pub Finder gets an inbound link and you start indexing all our pages, we get lots of hits, then a couple of weeks later our pages start dropping out of your index again. It's a little bit frustrating. And perhaps more concerning is the power you wield. But so long as you stick to your 'Do no evil' mission statement, I'm sure there isn't a problem, right?

Another way to get more hits to your site

I guess like most sites the Random Pub Finder gets most of its hit through Google searches. But a drunken chat with a friend led to another way to drive more traffic to the site. I've uploaded all the pub images to Flickr and added links to the reviews on the site. So far results have not been spectacular but it means we have deep links into some of our pages so their PageRank should improve. And who knows, it might work for you.

Build it and they will come

After five years, the Random Pub Finder has finally become an overnight success. Google seems to have started re-indexing pages, although there's still a random nature to its indexing. We were up to 300 pages indexed, now we're back down to about 150. But the main reason for the sudden surge in traffic has been appearing at Programmable Web. Thanks to that, we've been covered on a couple of other sites. We've never had so many hits.

Now I'm actually getting traffic here as well, thanks to this post. Seems I'm one of the top results on Google if you search for directmailchat. Clearly I'm not the only one suffering from this spam.

Google Sitemaps improvements

I'm not sure when these changes were introduced to Google Sitemaps but there are now some useful little graphs that tell you how often your site is crawled and how responsive your site is. Go to the Diagnostic tab and click on 'Crawl Rate' to see it. You can even tell Google to crawl more often if you like (not sure if their bot will actually take any notice of this request or not). Overall, I like it a lot.

Google have cocked up again

It looks like Google has cocked up its data update again. The PageRank updates it started a few days ago look like they are being rolled back, so the Random Pub Finder is back to PR 0. More worryingly, if I do a site: search, all URLs bar three are now shown as supplemental. But If I do a search for particular keywords, other pages on the site are returned and aren't marked as supplemental. Go figure.

How to have a reasonably successful website

I can't pretend to know a lot about making a website a huge success, none of the sites I've been involved with have been massively successful, but I've kind of worked out how to get a reasonable number of visitors to a site without any kind of massive outlay. Here are a few of my thoughts.

Don't spend a lot of cash

Websites are pretty cheap to run. Even if your website is your business, you can probably keep it running with no visitors and no income for a pretty long time. Blowing any money you do have on advertising before you're even sure your site is up to scratch is a sure fire recipe for disaster. Better to get just a few visitors in who'll give you a good idea if you're heading in the rifgr direction and you can fix any problems you have before too many people spot them.

I think advertising in the old media is pretty much a waste, it's very expensive and it's hard to see if it's been a success. You can advertise using AdWords for next to nothing, giving you the chance to see if it works at all and also trying out different advertising approaches.

Don't ignore SEO

Google is fond of saying that you should optimise for the end user, not the search engine. I can understand where they are coming from but essentially this is pretty much untrue. If end users can't find your site, whether the site has been optimised for users or not is pretty much irrelevant, they'll never see it. Chances are search engines will be the biggest source of traffic to your site, so of course you need to optimise for them. I managed to triple the number of visitors to the Random Pub Finder just by applying some simple SEO, all completely above board and not impacting on the end user at all. I found this free download a very useful read.

Design isn't too important

There are many examples of butt ugly sites being pretty damn successful (craigslist being the most obvious example) and plenty of examples of beautiful sites that have crashed and burned (boo.com comes immediately to mind). The fact is if the idea behind your site is appealling, people will come back. If the idea doesn't appeal, it doesn't matter how good looking it is, people will go elsewhere.

One company I worked for redesigned their site about 4 times in one year, in an attempt to get more conversions. I've no idea how much they spent in total but the effect on conversions was pretty minimal, so it was pretty much wasted cash. Every redesign brought a host of new problems that needed fixing, along with pages stored in search engines that no longer existed. Not only that but time spent on those redesigns could have been spent refining and improving what was already there.

Keep focussed

I've never really stuck to this advice (this site covers whatever I can be arsed to write about) but if you want to get people coming back to your site it sure helps if your site covers a single topic, or a few related topics.

Iterate and watch

I come from a software development background and I've always practised a kind of iterative development technique. And I use the same approach when developing websites. Make a small change, see if it works, move on. Unlike software development, when developing websites the 'see if it works' stage doesn't just mean 'make sure it runs OK'. it also requires looking to see how it has affected traffic, so the iterative cycle can be somewhat longer. and to see if it works, you need to have some data. The cheapest tools (i.e free) for this job are Google Analytics and Google Sitemaps, which tell you how many visitors you're getting and what keywords are driving them there, along with heaps of other information.

Take advantage of free advertising

There are plenty of opportunities to get more hits on your site that cost nothing. When I post to a forum and I'm given the opportunity to provide a URL I always do. Of course, posting randomly to forums just to get a link to your site is generally considered as spam and will likely cause more harm than good. Sites like Technorati and digg can drag in more readers and directories related to your site can help pull in yet more eyeballs.

Wait, and wait some more

I have no knowledge of the search engine algorithms but I'm pretty sure, all other things being equal, a site that has been around for 5 years will rank higher than one that's been around for 1 month. So if you just hang around doing nothing, your site should start to get some more hits. I've never really done anything to get hits to doogal.co.uk, but I now get about 200 visitors a week.

PageRank update

I've just noticed that Google seems to have updated PageRanks across my various sites. And finally The Random Pub Finder has got some PageRank! It's been PR0 for about a year, now we are up to PR3. My home page and this blog are both up, wahey!

More fun with <noscript> tags

The guys at BetterDeal wanted to know why their website didn't rank as highly as some of their competitors. Short answer is inbound links, which seems to be the prime ranking decider used by Google. To illustrate I did a link:www<dot>carbroker<dot>com<dot>au search (replace the <dot> with ., don't want them getting anymore inbound links), which is one of their competitors. The odd thing was that one of their inbound links came from a penis enlargement site. Penis enlargement doesn't have a lot to do with new cars, other than the obvious psychological enlargement for men who buy big red Ferraris. Looking at the source of the penis site (purely for research of course) showed up the reason. The link was hidden away in a <noscript> tag. Car Broker are also the guys who have <noscript>betterdeal</noscript> hidden away in one of their pages slagging off reverse auctions, which is why they rank pretty highly for a search on "BetterDeal reverse auction". Although Google know about this, they haven't done anything about it and their algorithm still doesn't seem to handle <noscript> tags too well...

A better link: search

I've known for some time that Google lies when I do a link: search to see which sites link to my sites. Even though I know of sites that link to mine, Google doesn't list them. So I was very pleased to discover that it's possible to use @: instead of link: and many more sites are listed, although it still doesn't show some sites I know link to mine.

Not Google site: bug

So after looking more closely at what I thought was a bug with Google, it turns out it's not. site:http://www.randompubfinder.com/ returns only the pages in Google's main index, site:http://www.randompubfinder.com returns all pages, including supplemental results. What are supplemental results? Basically pages that might show up in search results if Google can't find anything better. So we now have 4 pages in the main index! Unsurprisingly our hits are way down, since most of our traffic is driven by Google. So why did this happen? It seems that Google now won't store all pages in the main index if your site isn't popular enough. By popular, I mean having lots of inbound links. We don't so we get the big heave-ho. Doing searches now that used to bring up links to us now actually shows less relevant results, so I'm not entirely convinced this is helping improve the user's experience. Somebody at Google has written a fairly lengthy article about the latest updates there, with lots of unhappy people leaving comments. Here's a blog post from somebody in a similar position to ourselves.