0 Comments

 

Here are some quick steps for how to whitelist or blacklist the Ahrefs bot or crawler if you are using Cleantalk.org on your site for spam prevention.

The issue that I had was that I wanted ahrefs to crawl my site (outdoorgearreview.com) using their Site Audit function and it returned with an error saying:

The HTTP server returned error 403: “Forbidden”. This request was likely filtered by server configuration.

There are a couple of ways around this: 

  • Deactivate the cleantalk.org plugin while the crawl is running
  • Add the following code to your robots.txt file: 

User-agent: AhrefsSiteAudit
Allow: /

  • Or filter the ahrefs bot in cleantalk… see steps below

Step 1:

Login To Cleantalk.org and click “Personal Lists”. 

cleantalk ahref step 1

Step 2: 

Scroll down and click “Add New Filter”

cleantalk ahref step 2 add new filter

Step 3: 

  • Select “Ip Network”  – Note: Do not select IP Address
  • Enter in IP network range for Ahref Crawler: 

54.36.148.0/24,
54.36.149.0/24,
54.36.150.0/24,
195.154.122.0/24,
195.154.123.0/24,
195.154.126.0/24,
195.154.127.0/24

  • Make a note so that you know what you are either blacklisting or whitelisting
  • Select the site you want to activate this for. Else select all sites. 
  • Choose whether you want to blacklist or whitelist

cleantalk ahref step 3 filter settings

That’s it. I hope this helps. Comment below if you have questions. 

Share: