alltio.
Blog · Crawlers

Most publishers that block Claude's search crawler are almost absent from its answers. The setting belongs to whoever runs the server.

By Carl Gunell · October 29, 2026 · 5 min read

Muck Rack's October 28 data ties Claude citations to one line in a robots.txt file, a line written per domain by the team that administers the host rather than per work by the rights holder.

Card showing a robots.txt rule, User-agent: Claude-SearchBot, Disallow: /, above a large orange 81%: of major publishers blocking it, almost no Claude citations.  Allowing it: 12%.  Set per server, by whoever runs it.
Among major publishers, 81% of those blocking Claude-SearchBot got almost no Claude citations, against 12% of those allowing it. Alltio, 2026.

On October 28, 2026, Muck Rack and its Generative Pulse unit released the October edition of "What Is AI Reading?", built on more than 40 million AI responses and 207 million citations logged in 2026, plus 1,020 unbranded prompts run in September across ChatGPT, Claude, Gemini, and Perplexity. One finding turns on a single configuration choice. Among major publishers that block Claude's search crawler, 81% received almost no Claude citations, while among those that allow it the figure was 12%.

The release presents that as an observed pattern, and Muck Rack itself cautions that citation behavior may shift as models are updated or retrained. It does not establish that the crawler setting alone produced the gap, and the methodology sits in a full report that I have only seen summarized in the release. Even read narrowly, the association describes how presence inside one chatbot is now being allocated.

Three agents and one file decide what Claude can reach

Anthropic's help center splits its web traffic into three user agents. ClaudeBot gathers material that may be used for training, and restricting it signals that a site's future content should be excluded from training datasets. Claude-SearchBot indexes pages for search, Claude-User retrieves a page when a person asks about it, and Anthropic says that disabling either may reduce a site's visibility in user search results.

That separation gives a site owner a real choice, since a publisher can decline training and still be found in search. The instrument is the robots.txt file in the top-level directory, which Anthropic asks site owners to edit for every subdomain they want to opt out. In many publishing groups that file is maintained by an IT, platform, or security team applying a policy written for bots in general, usually in response to server load and scraping incidents rather than to any judgment about a particular article.

The file speaks for the host while the answer draws on the work

A robots.txt directive describes paths on a server. It has no field for an author, a rights holder, or the terms on which a given piece may be used, and it has nothing to say once the same text is syndicated to a partner site, quoted in a newsletter, or archived at another address. A publisher can surface in an answer through a copy on someone else's host while its own directive turns the crawler away, or vanish because a platform team blocked a user agent it did not recognize.

This is the chatbot channel, where the answer is composed inside the app and a citation is the only visible trace of supply. Muck Rack measures that trace, and the trace carries no money. Labs meter AI compute tokens to bill the buyer of every answer, while nothing meters on behalf of the supplier of the knowledge inside it.

Europe placed its reservation on the same machine-readable layer

Article 4(3) of the 2019 copyright directive lets text and data mining proceed unless rights holders have expressly reserved their works in an appropriate manner, such as machine-readable means for content published online. Article 53 of the AI Act requires providers of general-purpose models to operate a policy that identifies and complies with those reservations and to publish a sufficiently detailed summary of training content, a duty that stops short of naming individual works. The Commission's code of practice for those providers points to robots.txt as one of the machine-readable signals a crawler is expected to honor.

So in a newsroom in Hamburg or Milan, the person who formally reserves rights on behalf of hundreds of contributors may be the engineer who edits a configuration file. That reservation covers a domain. A freelance specialist whose work appears on that domain, and again on a personal site with no file at all, holds two contradictory positions without having chosen either.

Library lending pays the authors who registered before the loans were recorded

Britain has paid authors for library use of their books since the Public Lending Right Act 1979, and the order of operations is fixed. An author, illustrator, translator, or narrator registers eligible titles with the British Library, which runs the scheme, and only published books with an ISBN that are registered by June 30 enter that year's calculation, with no retrospective payment for earlier years. Since July 1, 2025, the calculation has drawn on national loans data from UK public libraries, and the 2024/25 rate was 12.40p per loan, with annual payments between £1 and £6,600.

Nobody in that system decides who gets paid by changing a setting at the library. Registration names the work and the person entitled, the libraries supply the usage records, and settlement follows from both. AI has built the access switch at the server and very little of what comes after it.

The record has to name the work before anyone sets a switch

Alltio is the registry of works. A creator or publisher scans a work on their own device, the scan produces a fingerprint, and the record binds that fingerprint to the rights holder and a declaration of the terms on which AI may use the work. Registration is the provenance, and the declaration is made by the rights holder and tied to the work's fingerprint, whichever host happens to serve a copy. An Alltio record is not a US Copyright Office registration and confers no statutory remedy.

No payment is flowing today, and Alltio has no access to AI lab logs and cannot observe AI use. Attribution follows once AI labs provide the usage records that show which registered works were used, settlement is then metered in AI compute tokens, and an off-ramp converts those AI compute tokens to currency.

A crawler setting can keep a large newsroom in or out of an answer, and much of the long tail of specialist knowledge sits on hosts whose settings were never chosen with its authors in mind. Alltio delivers that long tail to AI labs as a collective, so a lab has one agreement rather than tens of thousands, against records that already state whose each work is and on what terms. Creators and publishers can register their work at alltio.com/register.

AlltioFingerprinted

Put your knowledge on the record.

Free during the founding period — fingerprinted in your browser; your file never leaves your device.

Create your record