Cloudflare is helping OpenAI find the long tail

On October 15, Cloudflare and OpenAI reported the first results of a research pilot that began in July. Signals from roughly 1% of Cloudflare's network helped OpenAI's search crawler discover and index hundreds of millions of web pages it had never seen. The examples the two companies chose are telling: the manual for one washing machine model, the current passport fees, and a local restaurant's latest menu, pages that can be useful even when almost nothing links to them.

The link graph never found these pages

Search was built on links. PageRank treated a link as a vote, so a page with no inbound links had no votes and stayed outside the index however good it was. A network that carries the traffic sees a different picture: it knows which URLs are requested, which resolve, which redirect to a homepage, and which have changed since the last visit. Cloudflare used those patterns to tell OpenAI where the real pages were, to filter out addresses that bots request but that don't exist, and to flag a single compromised page without condemning the whole domain.

OpenAI's interest in these pages has a documented basis. Kandpal and colleagues showed at ICML 2023 that a language model's accuracy on a factual question tracks how many documents about that fact appeared in its training data, and that retrieval at answer time narrows the gap for rare facts. A model can't learn the long tail from volume, because the long tail has no volume, so it has to fetch it.

Discovery identifies the page and the domain

What travels in the pilot is metadata. The companies were specific that a signal about a page grants no permission to fetch it, that the domain owner's settings still decide access, and that page content is not shared for model training. OpenAI ranks what it finds on its own terms, and neither company has reported how often the new pages were cited or how many visitors they received.

Everything the signal carries resolves to two things: a URL and the Cloudflare customer who controls the domain. For a manufacturer publishing its own manuals, that customer may also own the text. For an engineer writing on Substack, a researcher whose paper sits in a university repository, or a consultant whose white paper is mirrored on three sites, the domain belongs to someone else, and the person who wrote the work and holds the rights to it appears nowhere in the exchange.

Telecom solved the same identity problem, partly

In my experience, the closest precedent is caller ID. Robocallers spoofed numbers for years because the signaling system passed along whatever number the originating switch inserted. The FCC's answer, mandatory for large US carriers since June 2021, was STIR/SHAKEN: the originating carrier signs each call with a certificate issued under a governed trust framework and attaches an attestation level.

The levels matter here. Full attestation, A, means the carrier knows its customer and has verified that the customer is entitled to the number. Partial attestation, B, means the carrier knows its customer but has no verified link to the number, and gateway attestation, C, means the call merely entered the carrier's network. Terminating carriers feed these levels into blocking and traceback decisions.

Map the pilot onto that framework and the gap is easy to locate. Cloudflare is the originating carrier, and it knows its customer, the domain holder, well enough to vouch that a page exists on that customer's zone. The work is the number. Nobody in the exchange holds a verified association between the work and the person entitled to it, so the best the network can offer about any piece of writing is a B. A white paper mirrored on three sites would carry three such attestations, one for each host, and not one of them would name the author.

Who would sign the work

The signals operate at the platform level, which raises the question of how anyone would sign at the level of the work. Under STIR/SHAKEN the carrier can give an A because it assigned the number and keeps the record of who holds it. The authority comes from that record, and the signature only carries it. Cloudflare can sign anything it chooses, but a signature can only attest what the signer knows, which here is the zone and the account.

A work-level attestation needs three things the network does not have: a fingerprint computed from the work itself, so the claim survives when the text moves from one URL to another; the name of the rights holder bound to that fingerprint; and the terms under which AI systems may use the work. Whoever holds that record is the party able to attest to it, in the same way the number assignment record is what lets a carrier sign an A. Until such a record exists for a work, every discovery system, however capable, attests the host and leaves the author unattested.

What the pilot proves, and what it cannot meter

The pilot answers one question convincingly: a network vantage point finds pages that links never surfaced, and AI search wants them. The companies building these systems have now made the long-tail argument themselves. The next phase, by their own account, is to measure what happens after content is discovered.

Measurement needs a unit. The telephone network meters calls because every call carries a number and the CDR records it. An AI answer drawn from a newly discovered repair manual will be metered against the URL and the domain, because those are the only identifiers in the pipeline, and if the rights holder isn't recorded alongside the work before the meter starts, there is nobody to attribute the use to when it is counted.

Fingerprinted