When IAB Tech Lab announced the launch of the AI Content Monetization Protocols (CoMP) Working Group earlier this year (initially dubbed the ‘LLM Content Ingest API’ initiative), it came as a bit of a surprise to some in the industry.
The project is an ambitious one, with the aim of building technical plumbing fit for the new AI-directed era of the open web. But while developing common standards and shared protocols is Tech Lab’s bread and butter, the new working group isn’t directly concerned with programmatic advertising. Rather, the aim is to provide safeguards for publishers’ IP, and develop a framework for how they can license this content to AI companies in a quick, transparent, and fair manner.
Despite not being directly related to programmatic, Shailley Singh, EVP of product and chief operating officer at IAB Tech Lab, says the new working group fits squarely within Tech Lab’s remit. “Our overall vision is to make sure the societal benefits of the open internet are available to businesses and the creators and publishers who make content,” he said. “So if you look at the larger scope of our work, this fits right in.”
Publishers, who make up a significant part of the community which Tech Lab services, are already seeing their business models threatened by AI, with many reporting traffic drops as a result of growing adoption of AI tools. “If you lose traffic, you’re going to lose revenue,” said Singh. “And that affects the whole advertising ecosystem, not just the publishers. Because if publishers lose traffic, there’s less and less advertising overall, and so that affects SSPs, DSPs, and so on.”
The new working group, therefore, intends to help sustain the digital advertising ecosystem which Tech Lab has helped build. And, as it turns out, the group’s expertise in real-time data-powered trading could come in handy for the new tools it wants to build.
A door with a lock and key
The first of the new working group’s three aims is to give publishers a way to effectively restrict bots from accessing and scraping their content: an essential step if they’re to be able to monetise access to that data. Or as Tech Lab puts it, to give publishers a door with a lock and a key.
It will be the work of the group to figure out exactly how this works, but part of the job will be strengthening existing methods. For example, while robots.txt files are commonly used as a means of allowing or blocking crawlers, they’re often simply ignored. So mandating adherence to these files, and using web application firewalls on publishers’ content delivery networks (CDNs) to block any bots which continue to ignore robots.txt files, will be part of the equation.
Tech Lab will also work on making it easier to classify the different types of bots which visit publisher sites, and create a means for communicating with bots which do get blocked, to let them know what they need to do in order to be able to access the content they’re trying to reach.
In the wake of an announcement earlier this year from Cloudflare, one of the world’s largest CDN providers, that it was working on stronger means of blocking AI bots, questions were raised about whether the company would really be able to completely shut down bad actors. Singh says that there will likely be an arms race, as bad actors will try to find ways through the firewalls.
Part of the challenge comes down to figuring out the origin of bots and classifying them correctly. There are existing initiatives based on self reporting, but as Singh says, “why would you classify yourself if you’re a bad actor?” Building technologies which can categorise bots based on their observed behaviour, and then choose whether to block or report them, will be key.
One of the other issues frequently reported by publishers is that the terms and conditions set by big tech companies make it hard to block AI scrapers. If publishers want to be accessible on Google Search for example, they have to accept Google’s search crawlers. But then websites can’t opt in to Google’s search crawlers without also opting into its AI crawlers.
This issue, however, will be outside the CoMP working group’s remit. “That’s more of a contractual issue,” said Singh. “What we’ll do is provide the technology, but there needs to be a larger effort from publishers on the contractual side to try and set terms which work for them.”
Building an IP marketplace
Once publishers are able to properly protect their content, the next step is setting up a way for them to monetise it through licensing to AI companies. There are two key parts to this: helping publishers package their content in a way which AI companies are able to easily discover and ingest it, in order to decide what they want to pay for, and then to create a marketplace through which sales can be made.
These won’t be necessary for all deals. For big publishing groups and the major AI businesses, it will be easier to create large-scale agreements directly — of the kind we’re already seeing being agreed. But for publishers with less scale, or smaller and more niche LLMs, these sorts of deals might not really be viable. Singh added that the large deals struck by publishers won’t necessarily value their content fairly, since they’re not tied to usage, meaning large publishers might prefer to use real time trading tools instead. Hence the need for a framework to enable the automated buying and selling of IP access.
Starting to sound familiar? While the broad concept of real-time trading is similar to programmatic advertising, there are significant differences. For a start, with content licensing, the buyer makes the first move rather than the seller.
“Here it’s the reverse of our advertising model,” said Singh. “The buyer comes up with demand in real time, and then the sellers have to respond to that demand, so there’s no bidding.”
But other concepts from the world of programmatic could carry over. “There will be concepts like deal IDs which I think we can borrow,” said Singh. “So you could create a deal ID which states how many times they can crawl your site, how many times they use your content, with a price that’s agreed upon and floor pricing; we could also use some of the logging capabilities that we have with bidstream logs. So we can borrow some of those concepts, but we’ll have to tailor them to these directionally different transactions.”
Avoiding the doom cycle
For such an ambitious project, it’s important to have plenty of stakeholders from all sides of the equation involved. But at the time VideoWeek spoke with Singh, he said that the AI businesses by and large (aside from those with large ad businesses that IAB already engages with) hadn’t been playing ball.
“It’s very important that these companies engage, and they’re not engaging right now,” he said. “I’ve tried reaching out with several emails to the people I’ve been able to connect with, but nobody has responded at all, not even a ‘no’!”
But Singh emphasised that it’s in these companies’ own interests in the long run to take part, since a decaying open web would starve LLMs of content to scrape in the first place. “There’s a risk of creating a doom cycle for these companies,” he said. “If publishers start shutting up shop, the LLMs will get less quality content they can use, which ultimately will end up degrading the content they show to their own users. So it’s really important for them to get involved.”
Follow VideoWeek on LinkedIn.
