Essential Code?

December 2, 2008

Talking to robots-your little friends out there in cyberspace - may not be as mad as it seems. But take care, not every robot out there wants to be your mate. Google pays little attention to your meta keywords but, that said, it’s still worth including them if for no better reason than helping you plan out the content of your web pages. I’ve come across some websites that seem to have an entire constitution and manifesto set up for visiting robots-but if those website owners understood how Google operated a little better, it would quickly become apparent that a lot of these commands and instructions are pretty redudant. They’re better off removed. When the Googlebot lands on a page of your site it wants to index that page and then use the links to find out what happens next. So if you notice the command ‘robots,follow’ in your code, get rid of it, you’re just slowing the Googlebot down and confusing your message. The Googlebot was going to follow whether you invited it to do so or not.

No index/no follow

There might be a reason that you don’t want Google to concentrate its efforts on certain pages because they are (content wise) inferior to other pages on your site. If this is the case, then it is a good idea to employ ‘noindex’ or nofollow and allow the Googlebot to get to the money pages quickly and without hidrance.

Meta Revisit-After

Get rid of it - who are you tell Google when they should pay you a visit? The Googlebot will visit when it damn well pleases and you should be grateful it does. Just make sure that there’s something new for Google to index every time it visits, be it a new image, some text or a new page.

Caution

Some robots are written not to scan or index your site, but are simply unleashed on the web to cause you annoyance by clogging up the server, demanding too many pages too quickly or generally making a nuisance of themselves. This may well happen to your site, but i wouldn’t advise blocking access to robots just because of the threat of malicious activity. The benefits of being open and available to the Googlebot and all the other search engine spiders far outweighs the possibility of a denial of service attack.