News

Cloudflare Tool Helps 'Capture Value' from AI Site Scrapers

• By David Ramel
• 09/23/2024

Connectivity cloud specialist Cloudflare announced a new AI Audit tool that monitors and controls how AI bots interact with site content, part of the back-and-forth between content owners and AI scrapers, with the former wanting to get paid by the latter.

The new offering -- actually a set of tools -- is designed to track bot activity, identify potential misuse, and help site owners manage access to content by setting rules and restrictions to ensures that AI bots are used responsibly and do not negatively impact a site's performance or security. And all of that is about money.

It's part of the ongoing tug-of-war between sites and content owners who believe they aren't being compensated for all the their data being used to train AI models that generate money for others. Basically, the tool lets Cloudflare customers "capture value" from new AI scanning bots that don't drive value to a site like "helpful bots" do because users just might stay inside a search interface, for example, and not visit the site.

"We believe this poses a risk to an open Internet," Cloudflare said in a blog post today (Sept. 23). "Without the ability to control scanning and realize value, site owners will be discouraged to launch or maintain Internet properties. Creators will stash more of their content behind paywalls and the largest publishers will strike direct deals. AI model providers will in turn struggle to find and access the long tail of high-quality content on smaller sites."

The post describes in detail the five-step workflow:

1. Understand how AI models use your site
2. Give yourself a pause to decide what to do next
3. Control the bots you do want to allow
4. Audit your existing scanning arrangements
5. Prepare your site to capture value from AI scanning

AI Audit provides analytics to see how AI bots access content, which the company said can help sites better protect their rights when negotiating with model providers. In such negotiations, sites can get help in getting a fair price for scanning rights, the company said.

"We think that sites of any size should be fairly compensated for the use of their content," says the waitlist to participate in a beta for price-setting capabilities. "Cloudflare plans to launch a new component of our dashboard that goes beyond just blocking and analyzing crawls. Site owners will have the ability to set a price for their site, or sections of their site and be able to charge model providers based on their scans of those sections. We'll handle the logging and filtering so that you can focus on creating great content for your audience."

Besides the price-setting capabilities that are in beta, the functionality to review analytics and control bots is available now for Cloudflare customers in the dashboard AI Tab.

"AI will dramatically change content online, and we must all decide together what its future will look like," said Matthew Prince, co-founder and CEO, in a news release. "Content creators and website owners of all sizes deserve to own and have control over their content. If they don't, the quality of online information will deteriorate or be locked exclusively behind paywalls. With Cloudflare's scale and global infrastructure, we believe we can provide the tools and set the standards to give websites, publishers, and content creators control and fair compensation for their contribution to the Internet, while still enabling AI model providers to innovate."