The web is becoming a customs checkpoint for bots

Cloudflare’s old AI bot control had the subtlety of a nightclub bouncer with one answer: no. The new version is more interesting: site owners can now treat search crawlers, training crawlers, and user-directed agent crawlers differently, as The Decoder reported.

That sounds like a product settings update. It is really a statement about the web.

The obvious reading is that publishers want better tools to stop AI companies scraping their work. True, but too narrow. The deeper shift is that the web is moving from open access by default to purpose-based access. The question is no longer “can this machine read the page?” It is “what kind of machine is this, what is it doing, and who benefits?”

Search bots used to get a fairly privileged deal because they sent traffic back. Training crawlers take value without the same bargain. Agent crawlers muddy the water further because they may be acting on behalf of a real user. Blocking all of them as “AI bots” is too blunt. Treating them differently is the beginning of a new customs regime for the web.

The border is moving inward

Google’s version of the same story happens at the user level. TechCrunch reported that using Google can feed its AI training unless users opt out.

This is where the old consumer bargain gets slippery. People understand, roughly, that using a free product creates data. What is less intuitive is that an ordinary interaction can become part of an AI improvement pipeline. A query, a click, a correction: these are not only product events. They are raw material.

Cloudflare gives site owners a checkpoint. Google gives users a settings page. Both are attempts to draw a boundary around machine consumption after the machine has already become embedded in the product.

The economic parallel is customs classification. A port does not simply ask whether something is “goods”. It asks what kind of goods they are, where they came from, what they are for, and which rules apply. That classification creates power. It decides who pays, who waits, and who gets inspected.

The web is reaching the same stage. “Bot” is no longer a useful category on its own. A crawler for search, a crawler for model training, and an agent booking something for a user may all look similar in server logs. Economically, they are different species.

AI pollution creates AI policing

Then there is Reddit, which is using large language models to solve a problem that LLMs largely created, according to TechCrunch. The irony is obvious, but the product lesson matters more: once synthetic content becomes cheap, moderation has to become more semantic.

Earlier anti-spam systems could lean on repetition, links, velocity, account age, and known bad behaviour. LLM-generated junk can be more varied, more plausible, and more context-aware. So platforms reach for LLMs to spot the patterns. The machine creates a new class of waste; another machine gets hired as sanitation.

That is not hypocrisy. It is what happens when the cost of producing convincing text collapses. The scarce resource becomes trust: trust that a page was accessed for an acceptable reason, trust that a user’s activity is not being quietly absorbed into a training set, trust that a comment thread still represents human intent rather than synthetic pressure.

For product teams, this is no longer a policy footnote. If you are building agents, crawlers, search products, marketplaces, forums, or AI features inside existing apps, you are now designing for border control. You need to answer basic questions clearly: What data are you taking? For what purpose? On whose behalf? Can the owner say no? Can the user inspect or reverse the choice?

The open web is not disappearing. It is being reclassified. The next competitive advantage may be simple: machines that can prove why they are at the door.

---