Crawling and Indexing

  1. Improving Indexation

    Whenever users type a query, Google searches through its index (a massive database of all web pages) and retrieves the most valuable content.

    Read more
  2. Solving crawl issues

    In some cases, SiteGuru cannot access your website. In this article, we describe why that happens and what you can do about it.

    Read more
  3. XML Sitemaps

    XML sitemaps are a powerful tool in your SEO toolbox. This article explains everything you need to know about sitemaps

    Read more
  4. Indexed, though blocked by robots.txt

    Learn how to fix all the problems that could cause the Indexed, though blocked by robots.txt status in the Google Search Console in our detailed guide. Plus, make sure Google doesn’t index your private pages!

    Read more
  5. Robots.txt file

    Use a robots.txt file to instruct search engine crawlers which pages to crawl, and which to ignore. A small mistake in a robots.txt file can have big effects.

    Read more
  6. Search Console Crawl Stats Report

    Fix your crawl, and everything else becomes a breeze. Discover how to use the Search Console Crawl Stats report like a pro and troubleshoot issues!

    Read more
  7. Noindex tags

    Even though Google’s bots constantly crawl the internet, that doesn’t mean they have to crawl and index every page on your website.

    Read more
  8. Redirects

    301, 302, 307... It’s time to demystify all the main redirect types in SEO! Learn everything you need to know in our redirect guide.

    Read more
  9. Crawl budget

    Don’t waste your crawl budget! Learn the top strategies for getting your pages indexed and ranking faster.

    Read more
  10. Removing Pages from Google and Bing

    Removing pages from Google and Bing is easy, but it's important to follow the right steps. This article tells you how to remove content from search engines.

    Read more
  11. Google Coverage Statuses

    Get indexed & get rankings! Learn everything you need to know about the 20 Google Coverage Statuses (and how to fix them)

    Read more