The World Wide Web is teaming with creepy crawlers commonly known as spiders. These spiders, also known as bots, crawl the web searching for food. What they like to eat is fresh content. So if a webmaster puts up something new, the spiders will soon be attracted to that content.

Web spiders aren’t really spiders. You know – the kind of with eight legs. Or was it six? Anyway, web spiders are computer programs created by the search engines to look for new data to index into their database. Since most web publishers want their data to be indexed and shared with the world through the search results, they try to make it as easy as possible for the spiders to crawl their site. This is why they create sitemaps. These maps of their website pages, navigation, links, linking structure make it easier for Google, Yahoo, MSN and the other search engines to make sense of the content on the web publisher’s site. Most importantly, sitemaps show the spiders how the data is connected. Spiders want to understand both data and linking structure so that they can determine relevance.

You can create a sitemap by simply listing your web pages in an outline form in a simple text file. I don’t recommend this as it is hard to maintain. In addition, you can’t be sure that your format will be understood by the spiders. It is best to go to Google, and use their sitemap standard. Alternatively, go here to create your sitemap online.

A robot.txt file tells the spider want they can index and what they cannot. The reason you would want to exclude certain files and/or directories from being indexed by the bots or spiders is for privacy. For example, you don’t want to show where you have stored the ebook that you sell on your site. If you don’t specify what to exclude, the robots will index everything that they come across on your site.

Here is a site where you can create your robot.txt file.