Why AI Web Agents Use Screenshots Instead of HTML

Yeah, why screenshot? So we actually put out a blog post on exactly this, that like why screenshot and not like the DOM and HTML. Our thinking was that the rendering of the website is very much intended for a human to consume it, but behind it is, right, is this machine code. Like there is the HTML, there is a DOM, so it makes sense for the machine to just use that directly. Like why use this rendering? And what we found over time is that different websites are built in such different ways that there is so much noise if you just sort of take that raw HTML information that it's very hard to train models to be reliable enough. But if you're trying to build web agents that are general, that can interact with websites just across the entire web, this is just not going to be scalable over time. And so what we found is that just using the screenshot is what gives you that generality.