robots.txt
Sometimes you have sections of your website that you don’t want search engines to index. That’s what robots.txt, AKA the “Standard for Robot Exclusion” was invented for. Now, 20 years later, there’s an article about common mistakes in using it.