Cloudflare AI Crawler Changes: September 2026 Guide
Review Cloudflare AI crawler settings before 15 September 2026 to protect Googlebot access, AI visibility and deliberate training controls.
Cloudflare will change its AI crawler defaults on 15 September 2026. The update separates automated traffic into Search, Agent and Training behaviours, but it also changes how mixed-purpose crawlers are handled. If a crawler is classified for both search and training, the most restrictive applicable policy wins.
That detail matters because Cloudflare specifically names Googlebot, Applebot and BingBot as mixed-purpose crawlers that can be blocked when a customer has chosen to block Training traffic. A setting intended to limit model training could therefore interfere with search crawling and reduce the freshness or reach of pages your business wants customers to find.
This is not another general debate about whether AI crawlers are good or bad. It is a deadline-led configuration guide for Australian business owners, marketing teams, SEO managers and technology decision-makers whose websites use Cloudflare. It explains what changes, what to decide, what to test and what to monitor after the cutover.
Three behaviours replace one blunt AI-bot choice
Cloudflare's policy model now separates how automated systems discover, use and act on your public website.
Search
Collects or indexes content for search indexes or retrieval-augmented answers. For most service and ecommerce websites, this is the behaviour most closely connected to discoverability.
Agent
Acts in real time on a person's behalf, such as a chat fetcher or browser-use agent. Blocking can affect customers who ask an assistant to visit a public page.
Training
Collects content to train or fine-tune models. A business may reasonably make a different choice for training than for search or user-directed access.
What happens on 15 September 2026?
Cloudflare says two changes take effect on the same date. First, new domains will default to blocking Training and Agent bots on pages that display ads, while Search remains allowed. Second, mixed-purpose crawlers will be evaluated across all of their classified behaviours.
Each behaviour can be set to Block on all pages, Block on pages with ads or Allow. If a crawler falls into more than one behaviour, the strictest applicable setting controls the request. A Search allow does not override a Training block for a crawler that Cloudflare classifies as both.
Cloudflare's legacy Block AI bots setting is also marked for deprecation on 15 September. Customers who want mixed-purpose crawlers treated differently from the incoming defaults can record that preference in Security settings before the date. Do not assume the legacy toggle expresses the policy your business wants after the migration.
Who should act first?
- Businesses that rely on Google or Bing for enquiries, bookings or ecommerce traffic.
- Sites that previously enabled the legacy Block AI bots option.
- Publishers or ecommerce sites with advertising on some page types.
- Teams with custom WAF, bot, redirect, transform or skip rules.
- Agencies and managed-service providers responsible for several Cloudflare zones.
Separate discovery, user action and model training
Choose the business outcome before the dashboard setting
A crawler policy should begin with the job each public page performs. A service page exists to be discovered. A product page may need both search visibility and real-time access by shopping agents. A paid research archive may have more value as protected intellectual property than as a source for model training.
| Behaviour | Typical benefit | Typical risk | Practical starting point |
|---|---|---|---|
| Search | Indexing, citations, AI answers and qualified discovery | Crawl load and reuse inside retrieval systems | Allow on public marketing, product and support pages unless there is a specific reason not to |
| Agent | User-directed research, comparison and future transaction journeys | Automated actions, scraping or unexpected load | Allow or monitor public informational pages; restrict sensitive, account or high-cost paths |
| Training | Possible long-term representation in model knowledge | Content reuse without a direct visit, licensing uncertainty and loss of control | Decide deliberately by content value and commercial policy; do not let a default make the decision |
This starting point is a business recommendation, not a universal technical rule. Document the decision, the accountable owner, the page scope and the date it should be reviewed. A service business may optimise for discovery; a publisher may prioritise licensing; an ecommerce operator may need agent access only on selected public catalogue paths.
Understand the mixed-purpose crawler trap
The most important migration question is not simply whether Search is allowed. Ask whether any crawler your business depends on also carries a Training classification and is therefore caught by the stricter setting.
Apple's own documentation shows why purpose separation can be subtle. Applebot is used for search experiences across Spotlight, Siri and Safari. Applebot-Extended is the separate robots.txt control that lets a publisher opt out of foundation-model training. Cloudflare's policy classification operates at the edge and may group a crawler by all observed or declared behaviours, so a robots.txt choice and a Cloudflare behaviour preset are not interchangeable.
For every crawler that matters to revenue or customer access, record:
- the operator and user agent;
- the Cloudflare behaviour classifications currently shown;
- the intended access decision for public and restricted paths;
- any separate robots.txt directive;
- the last successful request and normal response code;
- the test owner and escalation path if access changes.
Do not rely on user-agent text alone for a custom allow rule. User agents can be spoofed. Where available, use Cloudflare's verified bot metadata or Bot Management detection IDs, and verify crawler identity using the operator's documented method.
Audit every overlapping control
The dashboard policy is only one layer. Cloudflare explains that AI Crawl Control uses WAF custom rules, and that other rules can change the effective outcome. A crawler marked Allow can still be blocked by an upstream WAF rule. Skip, redirect or transform rules can also let traffic bypass a crawler block.
| Control layer | What to inspect | Failure to watch for |
|---|---|---|
| AI bot policies | Search, Agent and Training action for the zone | A strict Training action also blocks a mixed-purpose search crawler |
| Legacy Block AI bots | Current state and recorded migration preference | The old toggle is assumed to remain the source of truth |
| AI Crawl Control | Per-crawler Allow or Block actions | A dashboard Allow is overridden elsewhere |
| WAF custom rules | Rule order, verified-bot fields, paths, response codes and exceptions | Search bots receive a 403, or a skip rule bypasses a deliberate block |
| robots.txt | User-agent groups, wildcards and separate training tokens | A broad group contradicts the edge policy or blocks a crawler needed for discovery |
| Application and origin | Authentication, rate limits, middleware and hosting firewall | The edge allows a request that the origin rejects |
| Other CDN or security services | Bot protection, caching and proxy rules outside Cloudflare | A second control plane creates a result nobody can explain |
Keep private content behind authentication. Neither an Allow decision nor a robots.txt rule should expose account pages, internal documents, unpublished CMS previews or customer data to the public internet.
Run a crawl-access test matrix
Test the effective result, not just the saved configuration. Choose representative URLs from the homepage, a high-value service page, a product or category page, a recent article, an advertising-supported page and a path that should remain restricted.
- Capture the baseline. Export or screenshot the current Cloudflare policies, crawler actions, WAF rules and robots.txt file.
- Confirm normal browser access. Record the public response, canonical URL and any redirect chain.
- Test Google live access. Use Search Console's live URL inspection and confirm that crawling is allowed and the page fetch succeeds.
- Test Bing live access. Use Bing Webmaster Tools URL Inspection and review any crawl block, response or indexing error.
- Check crawler traffic. In AI Crawl Control, filter by crawler, operator, hostname, path and response status.
- Inspect WAF events. Look for unexpected 401, 403, challenge or redirect outcomes for verified bots.
- Run negative tests. Confirm that private or deliberately blocked paths remain inaccessible.
- Repeat after the change. Re-run the same cases after saving the new policy and again after 15 September.
A positive Google live inspection means the page can be crawled and parsed at that moment. Google cautions that it does not guarantee indexing or visibility. Keep monitoring impressions, indexed-page coverage and real crawler traffic after the configuration test passes.
A practical plan before 15 September
| When | Action | Evidence to keep |
|---|---|---|
| Today | Name the policy owner and list every Cloudflare zone, business-critical hostname and search crawler | Zone inventory, owner and business priority |
| Within 24 hours | Review the legacy toggle plus Search, Agent and Training settings; record an explicit mixed-purpose-crawler preference | Before-state export and approved decision |
| Within 48 hours | Audit WAF, AI Crawl Control, robots.txt and origin rules for conflicts | Rule map with expected response by path |
| Before cutover | Run Google and Bing live inspections and verify representative allowed and blocked paths | Test results, timestamps and defects |
| 15–17 September | Monitor bot response codes, crawl volume, Search Console coverage and Bing crawl errors | Post-change dashboard and incident notes |
| After one week | Compare the baseline, resolve unexplained 403s and schedule a monthly review | Decision log and recurring review date |
If the website is managed by an agency, hosting provider or former developer, confirm who controls the Cloudflare account before the deadline. A sound policy is useless if the responsible team cannot access the relevant zone or explain the existing rule chain.
Monitor business impact after the cutover
Cloudflare's AI Crawl Control provides request, bandwidth and response-code data by crawler, operator, hostname and path. Paid plans add referral metrics, while Bot Management customers can investigate detection IDs in Security Analytics. Use those views to find changed behaviour instead of waiting for a search-traffic decline to become obvious.
Track a small operating scorecard:
- allowed and unsuccessful requests for Googlebot, Bingbot, Applebot and chosen AI-search crawlers;
- 403, 401 and challenge responses by high-value path;
- Google Search Console page indexing and AI-search visibility trends;
- Bing crawl errors and live-inspection results;
- AI referral sessions and qualified conversions where available;
- origin load, bandwidth and cache behaviour;
- exceptions added, their owner and review date.
Treat a sudden block spike as an incident to investigate, not immediate proof that the new preset is at fault. Check WAF order, upstream rules, redirect behaviour, origin security and crawler verification before changing the policy.
Cloudflare AI crawler change FAQs
Sources checked
This article was prepared from current first-party guidance accessed on 9 September 2026:
- Cloudflare: Your site, your rules—new AI traffic options for all customers
- Cloudflare: Block AI Bots
- Cloudflare: Verified bots
- Cloudflare: AI Crawl Control with Cloudflare WAF
- Cloudflare: Analyze AI traffic
- Google: URL Inspection tool
- Apple: About Applebot
- Microsoft Bing: URL Inspection
Bot classifications, product controls and dashboard labels can change. Confirm the current Cloudflare documentation and your own zone configuration before applying production rules.