A helpful and effective tool for telling search engine crawlers how to index your website is robots.txt. Keeping this file under control is essential to effective technic...