Open web vs AI: what can W3C do?

At this week’s W3C Advisory Committee meeting, I ran a breakout session on what to do about threats to the open web. We had an interesting conversation. Many interesting points were raised, and some disagreed the web needs saving at all.

Disclosure: as the breakout was part of a Member-confidential event, I generalised and synthesised what was raised, and left out attributions to specific individuals, except where I explicitly asked permission (Members can view the minutes).

Wait, what threats?

Indeed, this session was inspired by earlier sessions at W3C and IETF. I’m not Mark Nottingham, but I really wanted this breakout to happen and discuss possible ways forward in the context of W3C, so I would just go and propose it.

Let’s first consider the problem at hand. A major reason for the web’s success is that it is open in many ways: it’s powered by open standards, it’s open for everyone to publish on (for fun, fame and/or profit), and it’s open for everyone to read. Perfect.

However… this “everyone” includes LLM crawlers, whose visits have been compared to DDOS attacks and frustrate publishers on the web, much more so than search engines have done for years. It causes websites from your average personal blog to Wikipedia struggle with server management and costs.

At the same time, content is increasingly consumed via LLMs and AI agents, which worries websites that care about content quality (like news organisations, governments, or public health information providers) and advertising income.

Comments from the blogosphere

On their blogs, various people added their thoughts to this discussion. Mike Masnick wrote about using AI assistants to create our own tools, and provide them with consent driven contexts via AT Proto. In Endgame for the open web, Anil Dash said we should remember how much the open web matters, volunteer at and support organisations that fight for it, like EFF and Mozilla, and consider what sort of policy or regulation could work. Julien Genestoux added we’re killing the web and recommended individuals to “publish in places you can leave”, meaning places where you can export data so that you can easily take it elsewhere. He also urged the community to rebuild social primitives, like identity, follow graphs, payment systems and discovery mechanisms, because existing primitives are provided by companies whose values don’t align with ours. Ben Werdmuller responded agreeing with Julien’s points, suspecting there’s interest in alternatives, and we need to engage one another more: “it is a work of engaged citizenry that verges on activism”.

Value extraction and end users

Chris Needham of the BBC takes part in the AI Preferences work at IETF and shared his worry about the value extraction that is going on, by AI companies, from companies that make the content, saying we must ensure fair value exchange. In that context, he mentioned the SPUR Coalition as one of the initiatives that will develop standards for responsible use of journalist’s content.

Not everbody agreed that the web “needs saving”, as I phrased it in my introduction, or that the problem is new: search engines have always scraped content from the web, in order to help users find that content. And we shouldn’t forget that a lot of end users want agents to help them browse the web, various attendees said. They can helps make sense of the enormous collection of information that is today’s web, in which it is increasingly hard to find stuff. Might AI agents may just qualify as Web User Agents?

What could W3C do?

In addition to AI Preferences (IETF) and standards for responsible use of journalistic content (SPUR Coalition), these are some things participants suggested W3C can do:

• Ensure AI companies are engaged in our standards work and community, possibly through workshops that are relevant for them (this topic is, as a web where nobody makes content will drive them out of business too);
• Building on that: serve as a bridge between AI companies and content creators… this is about publishers like the ones that are active at the W3C, but also much broader, including smaller content creators like bloggers or influencers;
• Take threats to the open web into account in work on WebMCP and the “agentic web”, making it an “open agentic web”, built on open standards and technologies (rather than proprietary) and based on open principles (rather than walled gardens);
• Consider a push model for content, so that websites don’t need to be crawled every second;
• Think about standards for payments (Web Monetization?) and attribution, complementing what SPUR will work on.

Summing up

I was glad to see this breakout full, and though we weren’t able to close out the queue before we ran out of time, we got a little closer to thinking about possible strategies. I am sure this conversation is going to continue in the next month, inside and outside of the W3C community. If you're reading this and have thoughts, do feel free to share!