(Reading time: 2 – 2 minutes)

I’ve just come across a great little tool to help you build a robots.txt file for your website:

Robots.txt Generator

What is a robots.txt file? It’s basically a little text file you upload to your server which tells search engines which pages or directories of your site you want them to spider (or not spider). In other words, it tells search engines which pages to look at (please) and which pages to please ignore and not list in the Search Engine Results Pages.

This can be more important than one might think. You have files in your server that are not real webpages and you don’t want to distract search engines by allowing them to inspect and crawl over these boring unimportant (for SEO) files. Examples of this are image files in your “images” folder, or WordPress plugins and themes, etc.

You might also have a file which you don’t want indexed for other reasons. Such as the download page for ab E-Book you are selling. You would want people to only find that page after they have paid. If it shows up in the SERPs, they can just find it by accident and get it for free (gasp)!

So here is the little tool you can use. If you have more questions – well, Google it. But also leave me a comment so I can add some more information later when I have some more time!

Anna Williams is a blogger, webmaster, photographer, and poet. She has lived and traveled in four continents, and currently lives in a forest, in the Pacific Northwest. Visit her personal blog, view photos, or read poetry. Anna offers website development and related services.