Cloudflare tests paid access controls for AI crawlers
Cloudflare introduced Pay per Crawl in private beta, allowing publishers to set a domain-wide choice to allow, block or charge AI crawlers through request headers and HTTP payment responses.
Why it matters: AI retrieval and training created a machine-to-machine licensing problem that robots.txt and one-off publisher deals did not fully solve.