Bots read your site: Cloudflare's rules for crawlers and my news collector with 120 sources

On October 16, Cloudflare described a system that watches how verified bots behave: proving who you are is no longer enough, you also have to behave on other people's sites. A business owner sees this from two sides. Search engine and AI service bots visit your site, and you decide who gets in. And if you collect competitors' prices yourself, your collector is a bot too, and the same rules apply to it. I built a collector like that for a consulting company in London in 2023, so I've been on both sides.

What came out

On October 16, Cloudflare introduced a trust band for verified bots. A bot proves its identity by signing its requests, and then Cloudflare watches two signals: whether the bot follows robots.txt per RFC 9309 and whether it slows down after an HTTP 429 response. One-off violations don't count; what matters is a repeating pattern. The scores are to be published on Radar. There is already a precedent: in 2025 Perplexity's bot was removed from the verified list for visiting sites that had banned collection and trying to get around the block.

A day earlier, on October 15, Cloudflare described a pilot with OpenAI. Since July 2026, signals from the Cloudflare network have helped OpenAI's crawler find pages that search didn't know about: roughly 1% of the network took part, and that was enough for hundreds of millions of new pages in the index. The reverse part interests me more. The signals weed out nonexistent addresses that bots request again and again, redirects to the home page and copies of the same content, and change signals tell the crawler when a page is worth re-reading. A signal doesn't grant access.

On October 21, DataReportal published its Digital 2027 analysis: web use through the browser is falling, and according to the authors, AI is not the only reason. At the start of October 2026, 6.15 billion people used the internet, a little over 5 billion of them opened the web, and the hundred most visited domains even grew their audience by 4.5% over the year.

On October 22, Pew Research Center sent out its newsletter The Briefing on where Americans get their news in the digital age. I'm not taking numbers from it, but the theme is the same: readers get news in a feed and through an assistant, and reach the publisher's site less often.

And a practical example. On October 13, n8n published a guest post by a You.com engineer about building a morning news digest. Search by topic, then the Jev model from TypeSafe AI scores each article for about four hundredths of a cent: is there an event, is it about the right topic, and does the team care. Only what passes goes to You.com Research, which writes a brief with citations in Slack. A morning costs one to four cents, and on a quiet day nothing gets posted. Without deduplication, the author writes, one story sits in the digest three days in a row, and after three weeks people stop opening it.

Why this matters to a business

Take a car rental company. It helps to know competitors' prices for the same dates. The fastest way: a script that opens ten sites once an hour and collects the prices. If it doesn't identify itself, doesn't read robots.txt and doesn't slow down after a 429, it will first get blocked by Cloudflare and then land in a low trust band. The prices will stop arriving right when they're needed. The takeaway: a bot that collects data for a business has to follow the sites' rules, or it gets blocked and the data disappears.

The other side. A manufacturing company put its catalog and manuals on its site. Close it to every bot in a row, and those pages will show up neither in search nor in assistant answers. Open everything, and part of the server's capacity goes to requests for files that don't exist. You have to decide bot by bot and section by section.

My experience: Scraper Hub with 120 sources

Toward the summer of 2023 I started building a news collection service for a consulting company in London, Scraper Hub. Many of the materials they needed lived on regular web pages with no XML feed. I set up a virtual machine in Google Cloud with Debian, wrote all the collection logic in Python, the server side in Node.js and the interface, and served everything through Nginx. The user chose what to collect, everything at once or individual blocks, and watched the progress over WebSockets. In total the service collected material from 120 sources, and both a model and people wrote notes from what came in.

What I took away from that work: some sources have a feed, some have a page that updates once a week, and the markup moves without warning. So the collector has to remember what it has already seen and not pull the same page twice. That's what Cloudflare calls change signals and n8n calls deduplication. In the same collaboration I built Quote Finder: Puppeteer and headless Chrome opened an article, found a quote, highlighted it and took a screenshot. To the site, a browser like that looks like a regular reader and pulls in scripts and images, so you need to run it less often.

What I would do

For a collector you commission: an honest User-Agent with an address the site owner can write to. Read robots.txt. On a 429, stop for the time specified. Store the page's modification date and don't download what hasn't changed.

For filtering what you collected: as in the n8n example, first a cheap check, is there an event here and is it about us, then an expensive model for the digest. We didn't have that chain in Scraper Hub in 2023; today I would start with it.

For your own site: look in the logs at which bots come and where they go, allow what you need in robots.txt and close off the service sections, and put a rate limit with a 429 response on heavy pages like search and filters.

How to check what a model has parsed, I described in the article AI in data processing. The approach to collection: Python for documents and data and AI with result verification; the project itself is in the cases Scraper Hub and Quote Finder. From other reviews: what I did on the site for ChatGPT and Perplexity answers and why I build on free tools; earlier I wrote about how to measure brand recommendations in AI. If you need to collect data from sites or sort out the bots on your own, tell me which sources and what to do with them: Discuss your task.