Cloudflare Blocks AI Bots: A “Too Little, Too Late” Effort to Curb Scraping
Cloudflare will now block AI crawlers from accessing website content without permission, a move impacting AI model training. This policy, effective immediately for new domains, allows website owners to choose whether to allow AI scraping. Furthermore, Cloudflare will introduce a “pay per crawl” model, allowing publishers to charge AI crawlers for content access. As a major content delivery network, Cloudflare’s influence on roughly 16% of global internet traffic means this policy could significantly alter how AI companies obtain data.
John Q. Hosedrinker