SEMrush Access Requests and Unblocking Requirements

SEMrush is blocked by default on Straightsell websites when it uses the standard SEMrush user agent. This is a deliberate security measure designed to prevent unknown third parties from running large crawls that can create unnecessary traffic spikes, trigger protection rules, or affect site performance.

This article explains why SEMrush may be blocked, what customers need to do before Straightsell can allow it, and what limits still apply after access is granted.

Why SEMrush is blocked by default

The default SEMrush user agent is automatically blocked by our network. This helps prevent unidentified scanning activity from external users and reduces the risk of heavy audit traffic impacting website stability.

Based on previous support tickets, the most common issues are:

• SEMrush audits being interrupted

• 403 or crawl restriction errors caused by security rules

• False positives in audit reports when the crawl is partially blocked

• Repeated requests creating spikes in traffic

How to unblock SEMrush

Before Straightsell can unblock SEMrush, the customer must update the crawler settings in the SEMrush application.

1. Change the User Agent from the default SEMrush value to a custom name unique to the customer or agency

2. Set the crawl behaviour to respect robots.txt

3. Reduce the crawl speed to 10 requests every 10 seconds

4. Send Straightsell the exact custom user agent name that will be used

What Straightsell will do once confirmed

After the customer provides the required details and has confirmed that the crawler settings are set to the requested limits, Straightsell can review the request and apply the appropriate allow rule, typically in Cloudflare or related security controls, so the custom crawler can reach the site without being blocked by region or default bot restrictions.

This does not mean SEMrush is fully exempt from all protections. The crawl may still be affected if:

• it exceeds expected request rates

• it ignores robots.txt

• it behaves like abusive traffic

• it contributes to website instability

Important limitations after unblocking

Unblocking SEMrush is conditional, not permanent unrestricted access.

• Rate limiting may still apply

• Traffic may still be challenged or blocked if behaviour appears unsafe

• Robots directives must still be honoured

• Straightsell may remove access if the crawl affects website performance

This approach protects the customer website while still allowing legitimate SEO auditing.