Spiders, Crawlers and Bots Oh My!

Bots, Spiders, Crawlers are all nick-names for the little programs that are constantly scouring the Internet indexing every file they find. You can see which Bots have visited your Web site in your site’s stats.

For example, Googlebot is Google’s little indexing program. These Bots are how search engines become aware of your site and new additions/pages so you can be listed accordingly. Understanding how they work is pretty important to every site owner because you can control what they index.

If you don’t want to let these critters in to index certain areas of your site, (images, member or private directories for example) it is important that you set up a robots.txt file. These little algorithms, at least the well-behaving ones, will look for that file before they proceed in case you want to designate areas on your site/server that you do not want indexed.

Well-behaving ones? Yep, there are nasty little devils that ignore your robots.txt file requests. Shame on them!

Read up and understand what Webcrawlers, Webrobots, Bots, Crawlers and Spiders are all about @ http://en.wikipedia.org/wiki/Web_crawler.  Then, use that information to create your robots.txt file and control what information on your site is indexed.

You might also find these posts interesting:

1. RSS Feeds & Website Syndication- How to Attract The Search Engine Spiders and Increase Traffic to Your Website Product DescriptionIf you are a website owner or designer, RSS...
2. Don’t 4Get Your 404 Have you ever given any thought to your 404 page?...
3. Search Engines Bridging the Gap Between Internet Users and Web Pages Search engines acts as a bridge between millions of...
4. Become a Search Engine All-You-Can-Eat Buffet Everyone wants top 10 rankings as though just because they...
5. Become an All-You-Can-Eat Buffet for Search Engines Everyone wants top 10 rankings as though just because they...