Robots, crawlers, spiders or agents are programs which are used to traverse the wobbly world wide web automatically, taking note on which content is where. Search engines use these programs to index content for their indices, spammers use them to scrape content for their own sites, or even to crawl the web for email addresses to spam you even more. Understanding of how search robots work is an intrinsic part of SEO!

What is robots.txt?

Robots.txt is a file which, when placed in the root of a publicly available webserver, tells search engine robots and agents which content they can and cannot/should and shouldn’t access. It can be used to block access to members’ only directories for example, or pages which nobody should really be finding through search engines, or, as I was trying to do earlier today, block search engines from indexing pages which are a part of your affiliate system or associated with an affiliate ID.

What is an affiliate system?

Many e-businesses run affiliate systems, which can be described as an semi-automated process by which the e-business takes referrals from partner websites, and remunerates them for any transactions that result from the referral. Having worked in the online gaming industry for over 10 years now, I’m quite familiar with many of them, as they an intricate part of just about every online gambling business model.

Most affiliate systems use standard html links which contain affiliate ID parameters in order to track referrals from their partner or affiliated sites. Anyone who clicks the link which contains the affiliate parameters will be associated with that affiliate’s account. If they go on to purchase something from the site, the affiliate will take their share of that revenue.

Search engine optimisation and brand protection

This is leading somewhere, I promise!

Many companies, when taking on an SEO consultant, agency or in-house employee, go straight for the proverbial jugular. They want to target the big, juicy keywords which will drive mountains of good, converting, valuable traffic. Because of this, they usually overlook the basics, ensuring you’re dominating the search results for your brand names. Imagine, if you will, the panic in the office this morning, when after no more than a few hours, I spot a discrepancy in our brand term search results: an affiliate tracked URL is ranking in second spot, taking a nice bounty per referral as well as a share of any future revenue from any clients that came through that link!

SEO, brand protection and affiliate URLs

Whoever was in here before me, had not taken the time to ensure that search engines, and especially Google (ye olde search dominator) were not allowed to index URLs tagged with affiliate IDs. Due to the high volume of traffic this affiliate was sending through his affiliate ID, that URL got indexed for our brand name, and is still ranking in second place. I immediately submitted a removal request, and asked for a change to the robots.txt to ensure that affiliate parameters were blocked from this point forward… the reply: “what robots.txt?”

Robots.txt to block affiliate IDs

Given the fact the content/web developer people here hadn’t implemented one, and the urgency required to get this resolved, I quickly typed up a robots.txt file for them to upload… ran it through the nicely-provided-by-Google testing utility:

Allowed by line 5: Disallow:

But what about line 6 you stupid test tool? The one that says:

Disallow: /?affid=*

No amount of fiddling would get it to work! I tried and tried and tried. And then I tested a second URL, one I typed up myself, and not copied and pasted:

Blocked by line 6: Disallow: /?affid=*

And that is when it struck me… affid and affId are two completely, and utterly different things according to robots.txt… why? Because ROBOTS.TXT IS BLOODY CASE SENSITIVE!

What is SEO?

Search Engine Optimisation can be defined as the manipulation of a website in order to increase its relevance towards a given keyword or set of keywords. However, as the online space evolves, more and more “social” elements can also have an effect on a website, its performance, and ultimately, its levels of traffic and conversions.

What can affect SEO?

Any online, and in many cases offline, promotional activity could have a positive, or indeed negative effect on a website. This includes:

• Editorial – Online or offline
• Advertorial – Online or offline
• Other forms of press – Especially online
• Social Networking Activity – Do your users, clients, staff, associates use social networks? This could impact your SEO perfomance
• Blogging – As above

As I’ve mentioned before, if you’re looking for SEO in Blackburn, please leave a comment and I’m sure I can point you in the right direction!

I stumbled back across the quiz a few weeks ago, and have been meaning to show off my latest result, so here it is:

SEO Dark Lord – 99%

Are you an SEO Expert?

Not having thought of this before, imagine my surprise to find that my blog has started ranking for “SEO Blackburn“.

So, as a concious blogger who knows that people finding my site whilst searching for a Blackburn SEO might not find what they’re looking for at this blog, please leave a message here and I’ll be sure to pass your information on to someone who would be able to help with any or all of your SEO requirements!

Do I really have to delve into any further detail here? Matt Cutts, seeing as you’re even mentioned in the article, I would LOVE your feedback!

• To find relationships between terms (synonymy¹ and polysemy²)
• To compare the documents contained within an conceptual space (document classification³)
• To translate a given query of terms into a conceptual space and find matching documents (information retrieval⁴)

It is the the use of LSA/I in information retrieval that really interests us here at SEO central, or wherever you may be. The fact that Google, and quite probably, but not to the same extent, the other major search engines take LSA/I into account when their algorithm performs a search can be the difference between your site ranking 1st or 101st for your specifically targeted keyword. The algorithms “know” that a page written on “baking” should naturally contain information on flour, eggs, sugar, cakes, bread, muffins, cookies, etc, etc… If you have merely blasted your page full of “baking” references, and not taken the other subjects that should at least be touched upon, then your page will be classed as overly optimised and won’t rank at all.

What this means for bloggers like you and I, is that the LSA/I segments or even subroutines of the search algorithms favour naturally written articles, that read well, and contain other relevant references, and seriously frown upon overly dense, over optimised, keyword-stuffed paragraphs that don’t read well at all.

¹The synonymic relationship between words. Synonymy is not always transitive, where a word can be a synonym of several other words that have completely different definitions. For example hot is a synonym of spicy and warm.
²Similarly, a polyseme is a word that has by definition more than one meaning. For example a mole is small borrowing animal, a skin marking, a spy or informant, the SI base unit for measuring the amount of any substance or even a large structure which is normally fabricated from stone, used as a pier or junction across water.
³The method or technique by which an electronic document is classified into one or more categories based upon the content of the document.
⁴The method by which a document is obtained or retrieved dependant upon it’s content in relation to a query.

Originally posted on this blog in July 2007, but was lost during the great blog explosion of 2008 and retrieved from Archive.Org Don’t ask why, but I’ve always missed this post and I’m glad to have it back! :)

I have recently noticed a lot of daily fluctuations in the SERPs for a reasonably competitive keyword that my client is targeting. One day it’ll go down 18 spots, then jump by 9, then again by 10, then down by 12, it’s all over the shop. Weirder still, we’ve not been changing much recently either, they just keep bouncing up and down. Then I spotted flagfox was showing a Dutch IP, which for a UK focussed site, isn’t great news.

I brought this up with the technical lead during our weekly call, and he mentioned that it could have been due to the load balancing system provided by Akamai. Dubbed as a “site accelerator” service, Akamai clients maintain one copy of their site on an “origin server” which then pushes the data out onto the rest of the Akamai server network. When a user wants to access a site hosted by Akamai, their global load balancing system defines and decides on the optimal server within the network to send the user too. All well and good if all the servers are hosted within the same country… not so all well and good when a UK facing site is hosted in the UK when the first Google DC hits in, in .NL the next time, .DE the time after that and .CO.CK the time after that (that’s the Cook Islands’ TLD, I swear).

<!– HTML Content for SEO –>

Under the comment are a few bits and pieces followed by a whole load of copy, held within tables that are hidden by CSS. Now, I’m all for usability/accessibility, but blatantly advertising the fact that the copy is included for SEO purposes is surely a no-no in anyone’s book?

He’s organising an SEO fest at Oktoberfest, with some of the biggest names in the SEO/Affiliate industries, with a number of jumbo parties thrown in as well.

Current situation I hear you say? Yes, I handed in my notice at bet365 as I have been offered a consultancy doing SEO/PPC, working from home, with a lot more flexibility, etc.

Fun times ahead, though I doubt SEOktoberfest 2008 will be on the cards as 5K is a LOT of money, even with a load of Bavarian beer thrown in.

Keyword Density Tool