Cloudflare and OpenAI Discovery: Website Audit Guide
A technically sound page can still be invisible to AI search if the crawler never discovers its URL. On 15 October 2026, Cloudflare and OpenAI published early results from a research pilot designed to improve that discovery step. Using signals from a participating cohort of about 1% of Cloudflare's network, the pilot helped OpenAI discover and index hundreds of millions of previously unknown pages.
The result matters to Australian businesses because valuable pages are often buried behind weak internal linking, inconsistent redirects, duplicate URLs, stale sitemaps or security rules that treat every bot alike. A service guide, product manual, support article or location page may be useful without attracting many conventional links. Better discovery can give those pages a chance to be evaluated for ChatGPT search.
It is not a new ranking shortcut. Cloudflare provides signals; OpenAI still decides whether a page is relevant and helpful. This guide separates the announcement from the hype and turns it into a practical website audit.
Why AI Website Discovery Is Trending Now
The pilot started in July 2026 and the first results were published on 15 October. The reported scale is the immediate news, but the more important point is architectural: AI search discovery can use network-level metadata to find pages that ordinary link discovery has missed.
Cloudflare and OpenAI say the pilot found a large body of useful, previously unknown pages while also filtering waste. Repeated bot requests for files that do not exist, sites that redirect almost every URL to the homepage and multiple copies of the same content can all create misleading signals. The pilot also tests change signals so a crawler can revisit a page when it is worth fetching rather than repeatedly downloading unchanged content.
For website owners, this shifts part of the AI-visibility conversation from content production to operational quality. Publishing another article is not enough if the site sends ambiguous status codes, blocks the wrong crawler, exposes endless URL variants or cannot show which public pages changed.
Three facts behind the announcement
Cloudflare and OpenAI published the early pilot results on 15 October 2026.
15 Oct
The participating cohort represented about 1% of Cloudflare's network.
~1%
Previously unknown pages were discovered and added to OpenAI's search index.
Hundreds of millions
Trace the path from policy to business result
What the Cloudflare and OpenAI Pilot Changes
Traditional crawlers often learn about pages through links and sitemaps. The pilot adds another discovery source: signals observed at Cloudflare's network edge. These can indicate that a page exists, has changed or is probably unhelpful to fetch. OpenAI then independently evaluates discovered pages and decides how, or whether, they appear.
This can help websites with useful long-tail material. A current product specification, exact service eligibility rule, obscure support answer or local operating detail may solve a real question even when few other sites link to it. More efficient discovery gives the crawler a better candidate set without asking it to fetch every URL more often.
The pilot also has a safety dimension. Cloudflare says it is testing signals that can flag potentially harmful pages, including a compromised page on an otherwise legitimate domain. Page-level treatment is important because excluding an entire business website for one malicious URL would be unnecessarily blunt.
The practical change is therefore not a new markup tag or CMS plugin. It is evidence that technical discovery, page quality, security and crawl governance are becoming one connected discipline.
What the Pilot Does Not Promise
The announcement does not provide a public switch that places a website into the participating cohort. It does not say that Cloudflare customers receive a ranking boost, and it does not guarantee a citation, answer inclusion or visit. Discovery only supplies a candidate page; OpenAI evaluates relevance and helpfulness separately.
It also does not collapse search and model training into one permission. The joint announcement says the research uses metadata about the web for search and discovery rather than sharing page content for model training. OpenAI documents separate controls for its search crawler and training crawler.
Finally, the pilot results do not make conventional technical SEO obsolete. Internal links, working server responses, clean redirects, canonical URLs and truthful sitemaps still make a site easier for people and automated systems to understand. Google documentation is useful for this general web hygiene, but it should not be presented as evidence of OpenAI's ranking rules.
Treat OpenAI crawler roles as separate decisions
One blanket allow or block rule can produce an outcome the business did not intend.
OAI-SearchBot
Used to surface public websites in ChatGPT search. Allowing it makes pages eligible for inclusion but does not guarantee placement.
GPTBot
Controlled separately for potential foundation-model training. A site can allow search discovery while opting out of this use.
ChatGPT-User
Represents user-triggered assistant access. Govern it according to the site's customer experience, security and content-access policy.
Audit 1: Decide the Policy Before Editing robots.txt
Start with a business decision for each type of access. Which public content should be discoverable in ChatGPT search? Which material should remain available to customers but excluded from model training? Which authenticated, personal, licensed or operational pages should no automated crawler reach?
Then compare that policy with the live configuration:
- Fetch the production
/robots.txtfile and record the rules forOAI-SearchBot,GPTBotand relevant user-triggered agents. - Check that the CDN, WAF, bot manager and hosting platform do not silently contradict the file with a 403 response or JavaScript challenge.
- Validate requests against OpenAI's published search crawler IP ranges where the security platform supports verified-bot checks.
- Review page-level
noindexdecisions. A crawler must be allowed to fetch a page before it can read that directive. - Keep sensitive or private information behind authentication. robots.txt is a crawler instruction, not an access-control system.
OpenAI notes that changes to its robots.txt controls can take about 24 hours to flow through its systems. Record the change time and verify behaviour later rather than assuming the edit worked immediately.
Audit 2: Remove Crawl Traps and Ambiguous URLs
The pilot's most practical lesson is that noisy sites waste discovery effort. Test a representative set of service, product, article, support and location URLs together with old and deliberately invalid addresses.
- Return honest status codes. A removed page should normally return 404 or 410. A moved page should use a clear permanent redirect to its real replacement.
- Eliminate blanket homepage redirects. Redirecting every unknown URL to the homepage hides errors and makes nonexistent addresses appear real.
- Consolidate duplicates. Align redirects, canonical tags, internal links and sitemap entries around one preferred URL.
- Control faceted and search URLs. Filters, tracking parameters and empty search pages can create near-infinite low-value combinations.
- Test rendered content. A 200 response is not useful when the main text appears only after a blocked script fails or the page renders nearly blank.
- Investigate sudden new paths. Unexpected login pages, spam directories or altered templates can indicate compromise.
Google's crawl and canonical documentation supports these general practices. They improve the web's clarity and reduce wasted requests, but they are not a published recipe for ChatGPT rankings.
Audit 3: Make Important Pages Worth Discovering
Technical access earns a page an evaluation, not a favourable result. The page still needs to answer a real question with information that is specific, current and attributable to the business.
Prioritise pages that help a customer make or complete a decision:
- service eligibility, inclusions, exclusions and process;
- product specifications, compatibility, availability and support;
- location coverage, opening hours and booking conditions;
- current policies, fees, timeframes and document requirements;
- implementation guides, troubleshooting steps and maintenance advice;
- first-hand comparisons, case evidence and clearly stated limitations.
Use one stable public URL for each durable subject. Put the direct answer near the start, use descriptive headings, name the responsible organisation or expert, show a meaningful reviewed date and link to related detail. Avoid manufacturing thin location or product pages that differ only by a few words. Distinct information is useful; duplicated templates are noise.
For a broader content approach, see VaniTech's guide to making a website appear in AI results.
Audit 4: Connect the CMS Publishing Workflow to Discovery
Cloudflare and OpenAI describe change signals as a way to revisit pages when fetching is worthwhile. Website owners cannot assume access to a direct submission channel from the research pilot, but they can make the publishing system's public signals accurate.
Review the CMS and deployment workflow:
- publish stable URLs and avoid creating a new slug for routine edits;
- update sitemap
lastmodvalues only when the page changes materially; - remove deleted pages from navigation and sitemaps and return the correct status;
- add internal links from relevant hubs when a new page becomes important;
- invalidate CDN caches so crawlers and customers receive the current response;
- prevent preview, staging, parameter and print variants from becoming public duplicates;
- log publishing failures when the CMS says an item is live but the delivery layer still serves an old or broken version.
This is where CMS governance, hosting and AI visibility meet. Marketing owns the information and its freshness; developers own delivery correctness; the business needs a named person to verify the outcome.
Audit 5: Configure Cloudflare Without Blocking the Goal
Cloudflare customers can use AI Crawl Control to review crawler requests, unsuccessful responses and robots.txt violations, then allow or block individual crawlers. Advanced WAF rules can add path-level exceptions. The correct setup follows the policy defined in the first audit step.
Check four layers together:
- robots.txt: confirm that the intended directive is actually served on every relevant hostname.
- AI Crawl Control: review OAI-SearchBot activity and whether its action is Allow or Block.
- WAF and bot rules: look for generic automation rules, managed challenges or rate limits that override the intended access.
- Origin health: separate crawler policy failures from real 5xx, timeout, cache or application errors.
Cloudflare's free-plan detection relies on self-identified user-agent strings, while Bot Management plans can use detection IDs. That distinction matters when interpreting dashboards. A user-agent count is useful operational evidence, but it is not proof that every request is genuine.
Also review managed robots.txt settings before enabling them. A convenient default that blocks common training crawlers may be consistent with your policy, but the team must still verify that search discovery remains allowed where intended.
What to Do If Your Website Is Not on Cloudflare
The core audit does not depend on Cloudflare. OpenAI says any public website can appear in ChatGPT search when OAI-SearchBot can access it, but inclusion and placement are not guaranteed.
Ask the hosting, CDN or security provider to verify:
- the live robots.txt response and page-level indexing directives;
- access from OpenAI's published OAI-SearchBot IP ranges;
- 403, 429 and 5xx responses for crawler requests;
- bot challenges, geo restrictions and rate limits;
- canonical, redirect and sitemap consistency;
- server logs for high-value pages and invalid URL patterns.
You will not receive the Cloudflare-specific dashboard or know whether the research pilot has discovered a page through its network signals. You can still control eligibility, remove technical ambiguity and publish more useful pages. Those are durable improvements across conventional and AI search.
Measure Discovery Without Promising Rankings
Use a scorecard that separates inputs, visibility and business outcomes. Otherwise a team may celebrate crawler activity that creates no useful exposure, or dismiss valuable visibility because direct clicks are modest.
Technical inputs
- share of priority URLs returning the intended 200, redirect, 404 or 410 response;
- OAI-SearchBot success, block, challenge and rate-limit counts;
- duplicate and soft-404 issues resolved;
- time between a material publish and the correct public response.
Visibility indicators
- priority questions for which the brand or page is cited in repeatable monitoring;
- which page earns the citation and whether the answer is accurate;
- new and returning ChatGPT referral sessions using
utm_source=chatgpt.com.
Business outcomes
- qualified enquiries, bookings, assisted conversions and product discovery;
- engagement with guides, service details or support content;
- incorrect answers, stale citations and support demand caused by confusing content.
OpenAI explicitly says placement is not guaranteed. Treat prompt sampling as directional and connect referrals to meaningful outcomes. VaniTech's AI search visibility measurement guide provides a wider framework.
A Practical 30-Day Plan for SMEs
Week 1: Set policy and baseline
Choose the public content that should be eligible for ChatGPT search and the content excluded from training or automated access. Export priority URLs, crawler rules, sitemap entries, canonical tags and current ChatGPT referral data.
Week 2: Fix access and response quality
Test OAI-SearchBot access through the CDN and WAF. Repair 403 responses, soft 404s, redirect chains, blanket homepage redirects, duplicate URL families and broken rendered pages. Recheck after configuration changes have had time to propagate.
Week 3: Improve ten commercially important pages
Select pages tied to revenue or customer service. Add direct answers, specific evidence, clear ownership, review dates, helpful internal links and genuinely distinct information. Update the sitemap only when each page is materially improved.
Week 4: Verify and operationalise
Review server or Cloudflare logs, confirm successful public responses, record ChatGPT referrals and sample a stable set of customer questions. Assign monthly ownership for crawler policy, technical errors, content freshness and inaccurate AI answers.
The deliverable should be a short register of priority pages, live controls, unresolved risks, owners and metrics. That is more useful than a one-off robots.txt edit.
Questions website owners are asking
Final Recommendation
The Cloudflare and OpenAI pilot is a meaningful discovery development, not a guaranteed visibility feature. Its early results show that useful pages can remain unknown and that edge signals may help search systems find them with less waste.
For Australian businesses, the right response is a controlled audit. Separate search, training and user-triggered crawler policies. Fix blocked requests, fake-success responses, duplicates and stale publishing signals. Improve a small set of pages that answer real customer questions, then measure crawler success, citations, referrals and business outcomes independently.
If the website sits behind years of CMS changes, CDN rules or inherited security controls, treat this as technical takeover work. The most valuable first step may be learning what the site actually serves, not adding another AI-search plugin.
For related governance decisions, read Should You Block AI Crawlers? and VaniTech's guide to Cloudflare AI crawler controls.
Sources Checked
- Cloudflare and OpenAI — Making AI discovery better for everyone
- OpenAI — Overview of OpenAI Crawlers
- OpenAI — Publishers and Developers FAQ
- Cloudflare — Manage AI crawlers
- Cloudflare — Directives and robots.txt monitoring
- Cloudflare — AI Crawl Control bot reference
- Google Search Central — Troubleshoot crawling errors
- Google Search Central — URL canonicalization
- Google Search Central — Sitemaps overview
Sources were accessed on 16 October 2026. The Cloudflare-OpenAI program is a research pilot and its cohort, product controls and search behaviour may change. Confirm current documentation before implementation.