AuditTagBot
If AuditTagBot appeared in your server logs, someone ran an SEO audit on your site. This page explains exactly what it did.
What it does
It fetches public HTML pages, plus robots.txt and sitemap.xml, and reads the markup for on-page SEO problems: titles, meta descriptions, headings, image alt text, canonical tags, structured data and social tags.
What it does not do
- It does not log in, submit forms, or follow links behind authentication.
- It does not execute JavaScript, so it will not trigger anything a plain HTML fetch would not.
- It does not write anything to your site, ever. AuditTag's entire premise is that fixes are handed to a person as files.
- It does not crawl the web at large. It fetches pages when somebody asks for an audit of that address, when a site owner has signed up for monitoring, or for a small, published research sample (see below).
Research scans
Occasionally AuditTagBot reads the public homepage and a few pages of stores in a defined research sample, to count which tracking tags are installed. Those scans follow the same rules as everything on this page, fetch at most ten pages per site, and are published only as aggregates, with no store named. The research methodology explains exactly how. To be left out, block the crawler in robots.txt or email us.
How much it fetches
At most twenty-five HTML pages per audit, four at a time, with a twelve second timeout per request. It skips images, stylesheets, scripts and documents. A full audit is typically fewer than thirty requests over a few seconds.
robots.txt
It reads robots.txt before crawling and honours Disallow rules for AuditTagBot and for *, using longest-match precedence. To keep it out entirely:
If a firewall is refusing AuditTagBot
Some firewalls and bot filters answer an automated visitor with a refusal or a check page. When that happens during an audit, the report says your site refused the crawler and links here. To let AuditTagBot in, allow it by its user agent, which is the name in the box at the top of this page.
- Find the tool that is refusing it. It is usually a firewall, a bot filter or a security plugin. Your server logs show the requests as AuditTagBot, and a response code of 401, 403 or 429 means the request was refused.
- Add an allow rule for requests whose user agent contains
AuditTagBot. Put it above any rule that blocks bots, because most tools read their rules from the top. - Run the audit again.
In Cloudflare
Open the Security rules page for your site, choose Create rule, then Custom rules. Use this expression, choose the action Skip, and under WAF components to skip pick the parts that are stopping the crawler. Place the rule above the rules that block. Cloudflare's page on the Skip action lists the options. That page also says Bot Fight Mode, on the Free plan, cannot be skipped this way (read on 30 September 2026). If that is the setting refusing the crawler, the rule will not help and you would need to turn that setting off.
Anyone can send a request that calls itself AuditTagBot, so this rule lets through anything using that name. Keep it narrow, for example by limiting it to the pages you want checked, and switch it off when you have no use for it.
On Shopify
Shopify runs the firewall and bot protection for your store. Its help page on bots says Cloudflare is always active on your store and needs no action from you, and it describes no setting for allowing a particular crawler. If an audit of your Shopify store is refused, email hello@audittag.com with the address and what the report said, and ask Shopify support if you want a crawler allowed.
About IP addresses
I do not publish IP addresses for AuditTagBot, so a rule that needs one cannot be written from this page. I could not confirm from Cloudflare's documentation, read on 30 September 2026, whether the crawler's requests leave from a fixed set of addresses, so I am not going to say either way. Match on the user agent instead.
If the report says your robots.txt blocked the crawler, that is a different thing. See robots.txt above.
Blocking it at our end
If it is behaving badly, or you want it blocked at our end rather than yours, email hello@audittag.com and we will add your domain to a permanent skip list.