# robots.txt

Sometimes you have sections of your website that you don’t want search engines to index. That’s what robots.txt, <span class="caps">AKA</span> the “Standard for Robot Exclusion” was invented for. Now, 20 years later, there’s an article about common mistakes in using it.

- [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
- [http://www.beussery.com/blog/index.php/2014/06/robots-txt-disallow-20/](http://www.beussery.com/blog/index.php/2014/06/robots-txt-disallow-20/)