Fix Cloudflare issues

Cloudflare is a service that sits in front of your website to make it faster and protect it from bad traffic. One of its security features, Bot Fight Mode, sometimes blocks our crawler by mistake. When that happens, pages or links show up as errors in your report, even though they work fine for real visitors.

The fix is to tell Cloudflare that our crawler is allowed. The steps below start with the option that works on every Cloudflare plan, including the free one. Use the first one that fits your setup.

Why our crawler gets blocked

We're a verified bot, so Cloudflare recognizes us. On paid plans you can use that status to let us through automatically. On the free plan, Bot Fight Mode can still challenge our crawler, and it can't pass that challenge. So the request fails, and the page or link looks broken in your report.

Allow our IP addresses

This is the most reliable fix, and it works on every Cloudflare plan. You add our crawler's IP addresses to Cloudflare's allow list, and it stops challenging them. The rest of your bot protection stays on.

Here's how:

1. Log into your Cloudflare account and pick your site.
2. Go to Security, then WAF (Cloudflare's firewall settings).
3. Open the Tools tab.
4. Under IP Access Rules, add each address below with the action set to Allow and the zone set to your site.

Add all of these addresses:

• 54.78.126.239
• 52.17.9.255
• 34.246.125.226
• 3.248.73.25
• 34.253.76.180

These are our current crawler IPs. We keep this list up to date, so if the problem returns later, check back here for any new addresses.

Or turn off Bot Fight Mode

If you'd rather not manage a list of IP addresses, you can switch Bot Fight Mode off. This is quicker, but it lowers the bot protection on your whole site, so the IP method above is usually the better choice.

1. In Cloudflare, go to Security, then Bots.
2. Turn Bot Fight Mode off.

Pro and Business plans: Super Bot Fight Mode

Paid Cloudflare plans use Super Bot Fight Mode instead of Bot Fight Mode. On these plans you can let our crawler through with a rule that matches its user agent. On the free plan this rule has no effect, so use one of the two options above instead.

First, check how our crawler identifies itself. In your Site Settings, select SiteGuruCrawler as the user agent.

Then add the rule in Cloudflare:

1. Go to Security, then WAF.
2. Click Create rule.
3. Set the Field to User Agent, the Operator to contains, and the Value to SiteGuru.
4. Set the action to Skip.
5. Under the components to skip, tick All Super Bot Fight Mode rules.
6. Click Deploy.

Check your country filters

Are you blocking traffic from certain countries? Our crawler runs from servers in Ireland, so make sure you're not blocking Ireland.

After you make any of these changes, re-run the report with the Check Again button. Still stuck? Our guide on solving crawl issues covers other reasons a site can block our crawler.