AI Crawlers

Introduction — The Evolution of Cloudflare’s AI Traffic Strategy Cloudflare’s 2026 AI traffic options introduce a fundamentally new approach to managing how artificial intelligence systems interact with websites. Instead of the binary “block all AI bots or allow everything” choice that defined earlier approaches, Cloudflare now offers a three-category taxonomy — Search, Agent, and Training — alongside new tools like BotBase for enterprise visibility, Signed Agents for cryptographic trust, and a critical September 15, 2026 default change that affects how multi-purpose crawlers like Googlebot, Applebot, and BingBot behave when Training is blocked. For website owners, publishers, and platform operators, understanding these options is no longer optional — it is essential for protecting content, preserving search visibility, and preparing for the agentic web era. ...

What Is llms.txt and Why It Matters for AI Agents in 2026 llms.txt is a plain-text file you place at your website root that tells AI agents and large language models which pages matter most. It’s the web’s first standardized machine-readable surface designed specifically for AI consumption — not for human visitors, not for search engines, but for the growing fleet of automated agents crawling the web. The format is dead simple: a markdown file with a brief site description, a list of essential links with one-line descriptions, and optionally a reference to an llms-full.txt that embeds the complete content of those pages. Anthropic proposed the standard in late 2024, and by mid-2026 it’s shipped by Stripe, Cloudflare, Vercel, Mastercard, ElevenLabs, and hundreds of other sites. ...