There are, broadly, two ways to teach a computer to use a computer.
The first is to hand it a pair of eyes. You take a screenshot, you pass the image to a model, the model says something to the effect of "I can see a blue button labelled Submit at approximately (842, 1190)", and then a harness obediently moves a virtual mouse there and clicks. Then you take another screenshot, because the model has no idea what happened, and you go round again. And again. This is the school of thought championed by Gemini and, in its most polished form, by Google's Antigravity: a larger model does the thinking & planning, delegates the actual poking to a smaller, cheaper, faster computer-use model, and that little model spends its short life screenshotting, clicking, and screenshotting again, like a very diligent tourist photographing every single step of a staircase.
The second is to admit that a webpage is not a photograph. It is a program, shipped to you in plaintext, with a fully documented interface for reading and manipulating every single thing on it. This is roughly the Claude approach: drive the browser through Playwright, read the accessibility tree, query the DOM, and — when the situation calls for it — run some JavaScript.
Both of these work. Only one of them feels like it was designed by someone who knew a browser was involved.
Reading my own exam results through a robot's eyes
I have used the first kind in fear, and I mean that fairly literally, because I used it to check my gaokao scores.
For readers unfamiliar: the gaokao is the Chinese university entrance examination, it happens once a year, and the results arrive through a provincial web portal whose information architecture appears to have been last revised for the Domesday Book. On results night, several hundred thousand teenagers hit that portal simultaneously, and it responds in the manner of a man being asked to carry a piano up four flights of stairs.
Also, I wasn't exactly willing to see the score myself — since I really did not know how I'd done. So I gave the job to Gemini in Chrome. I log in, deal with the captcha, and let Gemini click Query and read me the number.
It did it. Screenshot, coordinates, click. Screenshot, think, respond. It narrated the whole thing with the placid confidence of a pilot announcing a minor delay: "I will now click the 'Query Results' button." Mate. Behind that button is my entire adolescence. Perhaps we could show a touch of gravity about it.
It took something like forty seconds. In the taxonomy of human suffering, forty seconds is not much. It remains the longest forty seconds of my life, and I spent a meaningful fraction of it watching a mouse cursor glide across a screenshot of a government website, praying it wouldn't misjudge the y-axis.
But — and this is the point — it worked, on a site that no sane engineer would have written a selector for, with a captcha that exists specifically to defeat non-visual automation. The eyes approach is the lowest common denominator, and lowest common denominators are genuinely, unglamorously useful.
One hundred questions, ten tool calls
The second kind of agent I encountered under much lower stakes, and it impressed me considerably more.
After the gaokao, I was enrolled into HUST, where I had a knowledge questionnaire to fill in. A hundred questions. Multiple choice, radio buttons, the whole civic-duty aesthetic. I gave Claude access to the browser and went scrolling on my phone.
Under the screenshot-and-click paradigm, this task has a known shape, and that shape is a hundred-odd rounds of tool calls. Scroll. Screenshot. Locate question 14. Click option C. Screenshot to confirm C is now selected, because you cannot otherwise know. Scroll. Screenshot. Repeat until the heat death of your token budget. Each cycle is a round trip: model inference, tool execution, a fresh image dumped into context, more inference. You are paying, in both money and wall-clock time, for a hundred separate acts of looking.
Claude used about ten tool calls.
It read the page structure once, worked out that the questions were a uniform list of inputs with predictable names, and then wrote a small piece of JavaScript that selected all of them, thought about each of them, set the values, dispatched the appropriate events so the page's own framework would notice, and reported back a count of how many it had filled. When I asked it why it needed that many tool calls, Claude answered that most of the time, it was just furiously fighting against the weird DOM structure of the website. The actual clicking mechanism used only one tool call.
There is a version of this essay that treats that as a mere efficiency gain. It isn't. It's a difference in kind. The vision agent was solving a hundred problems. Claude noticed it had been handed one problem, repeated a hundred times, and solved it once. That is, more or less, the entire history of programming as a discipline, re-enacted in twenty seconds by something that does not technically know it is doing it.
The arithmetic, briefly
I don't want to turn this into a benchmark post, but the numbers deserve a moment, because they are not close.
A screenshot at any resolution a model can usefully read costs on the order of one to two thousand tokens. That is the floor. Now recall that agents carry their history with them: turn forty contains the transcript of turns one through thirty-nine, and if you're keeping the old screenshots, it contains those too. Context grows roughly quadratically in the number of steps, which is why every serious implementation ends up quietly evicting old images, which in turn gives the agent a rolling amnesia and a tendency to re-derive conclusions it reached four minutes ago. You can watch this happen. It's like observing someone re-read the same paragraph of a novel with genuine surprise each time.
Against this, the DOM route costs: one structural read, one script, one confirmation. The script itself is perhaps forty lines, and forty lines of JavaScript is cheaper than one screenshot — but it would be dishonest to stop the comparison there, because the script is the cheap part. The expensive part is the structural read that lets you write it, and an accessibility tree for a real page runs to many thousands of tokens, which is several screenshots' worth, and doesn't politely compress down to "there is a blue button here" the way a described image does. My ten tool calls were exactly this. One of them did the filling. The rest went on working out what on earth the page was made of.
The point survives the correction, because that cost is paid once and then amortised across a hundred questions, whereas the pixel loop pays its own cost a hundred times over. And what you get for it — this is the part I keep coming back to — is determinism. A querySelectorAll returns the same set every time you run it. A mouse click at (842, 1190) can absolutely miss, and it will do so silently, and then everything downstream is built on a lie.
On models being confused about the tools you gave them
Here is the observation that actually prompted this post.
Antigravity, to its considerable credit, will let you drive it with Claude instead of Gemini. The browser-clicking tool is still sitting right there in the toolbox. And Claude, given that tool, treats it roughly the way a chef treats a spoon you have thoughtfully carved out of soap: picks it up, turns it over, sets it down, and reaches for the knife it actually knows.
Gemini, by contrast, loves that tool. Reaches for it constantly and cheerfully. Which makes sense — if the pixel-and-click loop is what you were trained on, then coordinates are your native language, and the accessibility tree is a foreign document you'd rather not read.
I think this is the underappreciated dimension of agent design. We talk endlessly about model capability and almost never about model temperament — what a given model finds natural, what it finds baffling, and how much of its thinking gets consumed by the question "what on earth is this for?" Handing a model a tool that cuts against its instincts doesn't just waste the tool. It burns reasoning on the meta-problem of tool selection, which is reasoning the model is no longer spending on your actual task. A confused agent is an expensive agent.
Though temperament is probably too romantic a word for what is going on. What we are actually looking at is a fossil record of post-training: whichever harness got rewarded is the one the model reaches for first. That is the more boring explanation and also the more useful one, because it means temperament isn't a fixed property of a model but an artefact of the pipeline that produced it — and the mismatch I'm describing is less a model-selection problem than a tooling ecosystem drifting out of sync with the training that models are actually given.
The fat-finger problem
While we're being unkind, I should mention GPT-5.1, with which I had a period of browser-based cohabitation that I would describe as characterful.
Its distinguishing feature was the fat finger. It would decide to click a button, emit coordinates that were confidently and consistently about a dozen pixels off, hit the padding, and observe that nothing had happened. So far, fine — that's a miss, and misses are forgivable. What was less forgivable was the next inference, which was, invariably, that the website was broken. Not "my aim was off." Not "let me re-locate the element." No: the site was down, the button was non-functional, possibly the whole service was having an outage, and would I like it to try a different approach, such as reloading and doing exactly the same thing again?
In fairness to it — and I do mean this — asking a general-purpose language model to emit accurate pixel coordinates is a deeply strange request. These are systems that learned the world from text and were later shown some pictures. Expecting one to also function as a surveyor's theodolite is asking the poet to do your land registry. It's not that they can't. It's that we've built an entire interaction paradigm on top of the thing they're worst at, and then act surprised when the failure mode is spatial.
Gemini is the exception that rather proves the rule. It is noticeably better at this than anything else I've used, and the obvious explanation is that it was natively multi-modal from the beginning rather than having vision bolted on afterwards. Which is a genuine point in favour of the eyes approach — provided you bring the right pair of eyes.
In defence of eyes
Right. Having spent a thousand words admiring JavaScript, let me be honest about where it falls over, because the DOM route is not a free lunch.
Some pages genuinely are photographs. Anything rendered to a canvas — maps, design tools, games, an alarming quantity of enterprise dashboards — exposes essentially nothing useful to a query. Shadow DOM and cross-origin iframes will happily wall you out. Virtualised lists will cheerfully report six rows when the user can see six hundred. Some sites are actively adversarial: randomised class names, event handlers that check for plausible mouse trajectories, honeypot inputs that exist solely to catch things that fill in every field they can find. And the structural approach fails loudly and early, which sounds like a virtue until you realise that "loudly and early" means "at the first of the eleven steps," whereas the pixel agent, for all its clumsiness, will keep flailing forward and sometimes arrive.
And "fails loudly" only really covers the case where your selector matches nothing. The nastier case is the one where it matches the wrong set — a hundred radio buttons, ninety of them yours and ten belonging to a second form further down a page you never scrolled to. Nothing throws. The count comes back plausible. That is a quieter failure than any fat finger, because a missed click leaves the page untouched, whereas a confident wrong selector leaves it changed and looking finished.
There's a security asymmetry too, and it cuts both ways. A vision agent reads what a human reads, which makes it charmingly immune to the white-text-on-white-background prompt injection that a DOM-reading agent will swallow whole — though it remains perfectly vulnerable to injection a human would also read, which is to say a banner politely asking it to disregard its previous instructions. On the other hand, a vision agent's worst-case action is a click, while a JavaScript-executing agent's worst-case action is arbitrary JavaScript in an authenticated session, which is a rather larger blast radius. Although that radius is more a property of the harness than of the approach: hand the model a read-only query capability, or require that scripts be shown to a human before they run, and most of the asymmetry goes away. Power and exposure are, as ever, the same quantity measured from different ends.
What you can't engineer away is the gap between looking and acting. A vision agent, by the very nature of reading-then-clicking, is exposed to the TOCTOU problem Johann describes: the button you identified in the screenshot and the button your click eventually lands on are separated by a second or two of wall-clock time, and a page that rearranges itself in that window will cheerfully accept a click you never intended to make.
What I'll say for the structural approach is that it is auditable. When Claude proposed to fill my hundred answers, it showed me the script first. I read it. A bunch of lines, no surprises, and I could tell before it ran what it was going to do. Nobody has ever meaningfully audited a hundred sequential clicks. You can watch the flipbook afterwards and hope.
The bilingual agent
The right answer is obviously not to pick a side, which is a disappointing thing to conclude after all that, so let me be more specific about it.
The right answer is an agent that reaches for structure first and eyes last. Read the DOM. Look for the repeated pattern. Ask whether this is one problem a hundred times or a hundred problems once. If the page is a program, treat it as a program. If the page turns out to be a photograph — canvas, captcha, a provincial results portal held together with hope — then fall back to squinting at pixels, and accept the cost, because the cost of squinting is much lower than the cost of not getting the answer.
What we have instead is two camps, each with a strong prior about which of those the world is made of, and a tooling ecosystem that mostly reinforces whichever prior it was built on. Gemini thinks the web is pictures. Claude thinks the web is code. The web, irritatingly, is both, and mostly in the same tab.
My gaokao score arrived via a bunch of screenshots. My questionnaire was filled by forty lines of JavaScript. I remember the first one more vividly, but I'd rather have the second one doing my work — and it says something about the state of the art that in 2026 we are still, with great enthusiasm and enormous compute budgets, teaching machines to squint at documents they could simply have read.