PaPoo
cover

Text, not pixels, is the right instinct

What jumped out at me is not the “browser automation” part — everybody has a browser automation story now — but the choice to make the page legible to the model as structured text first. That feels like the right argument to make against screenshot-based agents, because screenshots are such a lazy interface for an LLM. You hand it a picture, then ask it to infer state, labels, and click targets from pixels it can’t actually reason about very well. It’s expensive and brittle. So yes, the pitch makes sense.

I’m a bit more cautious about the implied win, though. “Compact text” is great until the page gets weird. The whole approach depends on the visible DOM being a decent representation of what matters. If the site is canvas-heavy, hides controls behind custom widgets, or changes layout constantly, the neat handle-based model could start to feel less like a clean abstraction and more like another translation layer that needs babysitting. The article gestures at this with the screenshot fallback, which is probably the honest part of the design. Text-only won’t cover everything.

The part I’d actually want to try is the batch action tool. That’s the piece that sounds genuinely useful for agents: read once, act several times, then re-read the updated view in one round trip. If that works smoothly, it could cut down on the slow, chatty loop a lot of agent stacks fall into. A browser agent that can click and type without burning a screenshot per move is a better mental model than “vision in a loop,” at least for ordinary web apps.

I also like that they’re pushing the local/private angle. Running against the browser you already use, with existing logins, and keeping the connection on 127.0.0.1, is much more believable than yet another cloud relay that wants access to everything. That said, “private” here mostly means local transport, not magic security. The extension still sits in a very powerful place, so I’d want to understand its permissions and failure modes before trusting it with anything sensitive.

So my read is: the idea is sensible, and probably better than screenshot-first browsing for a lot of workflows, but the hard part isn’t the token count. It’s whether the structured view stays faithful enough across ugly real-world sites. That’s the test I’d care about.


Reference: Web Draw: Drive the live browser as compact text, without screenshots

同じ著者の記事