In a landmark move set to redefine the digital economy, Cloudflare has announced it will now block unauthorized AI crawlers by default, shifting the internet towards a permission-based model and empowering content creators to control and monetize their intellectual property in the age of artificial intelligence.
BANGKOK, July 3, 2025 – Cloudflare, Inc. (NYSE: NET), a leading connectivity cloud company, today initiated a fundamental change in how the internet operates, declaring it will become the first internet infrastructure provider to block AI crawlers that access content without permission or compensation by default. This pivotal decision effectively ends the era of unrestricted data scraping by AI companies and hands the controls back to the website owners, publishers, and creators who form the backbone of the web. The new policy establishes a permission-based framework, paving the way for a more sustainable and equitable economic model that aims to benefit both content originators and the future of AI innovation.
For decades, the internet thrived on a simple, symbiotic relationship: search engines would crawl and index content, in return driving user traffic and advertising revenue to websites of all sizes. This cycle incentivized the creation of the high-quality, original content that made the web an invaluable resource. However, the meteoric rise of generative AI has shattered this model. AI crawlers indiscriminately harvest vast amounts of data—text, images, and articles—to train their models and generate their own answers, often without sending visitors back to the original source. This practice has systematically stripped creators of revenue and undermined the incentive to produce original work, placing the future of a vibrant, open internet at risk. As Nicholas Thompson, CEO of
The Atlantic, bluntly stated, “For too long, the giant AI companies have built their businesses on training data for which they never paid, from websites they never even asked for permission to use”.
Cloudflare’s action is a direct response to this growing crisis. Leveraging its position managing roughly 20% of the world’s web traffic, the company is enforcing a new standard. Website owners can now explicitly choose which AI crawlers are permitted to access their content and decide how AI companies can use that data, whether for training, inference, or search functions. This move formalizes a trend that began in September 2024, when Cloudflare introduced a one-click solution to block AI crawlers, a tool that has since been adopted by over one million customers. Now, this protection is the default. Every new domain joining the Cloudflare network will be asked upfront if they want to allow AI crawlers, shifting the model from an “opt-out” to a mandatory “opt-in” system.
Matthew Prince, co-founder and CEO of Cloudflare, articulated the profound vision behind this change. “If we want the Internet to survive the AI era, we need to give the creators who make it possible the tools to control their content and forge a new economic model that works for everyone,” he said. “Original content is what has made the internet one of the greatest inventions of the last century, and it’s critical that creators can keep creating it. The Wild West of AI crawlers scraping content requires us to return power to the hands of creators, while still enabling AI companies to innovate. This is about protecting the future of a free and vibrant internet with a new model that works for everyone”.
This bold initiative has been met with a groundswell of support from a broad coalition of the world’s leading content, media, and technology companies, who see it as an essential step toward restoring a fair value exchange. Roger Lynch, CEO of Condé Nast, called the move a “sea change for publishers” that “sets a new standard for respecting content online”. He added that when AI companies can no longer take what they want for free, it “opens the door for sustainable innovation built on permission and collaboration”.
This sentiment was echoed across the publishing industry. Neil Vogel, CEO of Dotdash Meredith, stated, “We have long said that AI platforms must fairly compensate original publishers and creators for the use of our content. Now we can restrict access to our content to only those AI partners willing to strike a fair deal”. Renn Turiano, Chief Consumer and Product Officer of Gannett Media, which operates USA TODAY and over 200 local publications, emphasized that blocking unauthorized scraping is critical. “As our industry faces these challenges, we are confident Cloudflare’s technology will help combat the theft of this valuable intellectual property”.
Tech platforms that rely on user- and partner-generated content have also thrown their weight behind the new model. Steve Huffman, co-founder and CEO of Reddit, asserted, “Any platform on the web should have the right to decide who gets to use their content and for what. The entire ecosystem of creators, platforms, web users, and crawlers will be better for more transparent and controllable crawling”. Similarly, Bill Ready, CEO of Pinterest, committed to “building a strong internet infrastructure where content is used as intended, so creators and publishers can thrive”. The support extends to the music industry, with Boyd Muir, Chief Operating Officer of Universal Music Group, welcoming the initiative to “help address the indiscriminate, unregulated, and unauthorized scraping of intellectual property”.
The implications of Cloudflare’s policy extend beyond mere blocking, laying the groundwork for new economic frameworks. The company is actively developing solutions like a “Pay Per Crawl” beta program, which would allow content owners to directly charge AI crawlers for accessing their material. This creates a direct path to monetization and a market-based incentive for AI companies to license high-quality data. To support this new economy, Cloudflare is also enhancing transparency by offering AI bots more reliable ways to verify their identity, giving website owners granular control based on verified crawlers. The company is participating in the development of new public protocols to create a standardized way for bot owners to identify themselves.
This multi-faceted approach has even garnered support from within the AI community. Bill Gross, Founder & CEO of ProRata AI, proudly announced, “We are proud to be one of the first AI companies to participate in Cloudflare’s initiative to create a new permission-based model for the Internet. We believe that creators and publishers deserve fair compensation for the value they provide”.
The move is seen by many as a necessary correction to the “steal first, apologize later” ethos that has characterized parts of the tech industry. John Battelle, Co-founder and CEO of DOC, praised the decisive action, stating, “It takes courage and conviction for companies to break with decades of legacy practice”. Jason Kint, CEO of Digital Content Next, was even more direct: “Permission is the law when it comes to copyrighted content. Cloudflare’s move to use technology to ensure AI companies cannot skirt the rules is a major step in restoring a fair value exchange for trusted publishers and creators”.
Ultimately, Cloudflare’s decision represents more than a technical update; it is a philosophical stand for the future of the internet. It challenges the notion that all data is free for the taking and instead proposes a system built on consent, transparency, and mutual respect. By empowering creators to protect their work and enabling new, fair models of compensation, this initiative aims to ensure that the internet remains a place of vibrant, diverse, and high-quality information, preserving the very ecosystem that AI itself depends on to learn and evolve. As the digital world navigates the transformative power of artificial intelligence, this move may well be remembered as the moment the internet’s original creators reclaimed their stake in its future.
#Cloudflare #AICrawler #DigitalRights #Copyright #FairCompensation #AIethics #ContentCreators #FutureOfWeb #TechPolicy #DigitalEconomy #PermissionBased
