Technical SEO


The robots.txt file is a file placed in the root of a website that has crawl instructions for search engines. It can tell bots which parts of the site it should not index, and can also be used to points bots to the sitemap of the page.