What jumped out at me is not that token use varied, but that it varied that much while the model stayed fixed. Seventy-fold is the kind of number that makes you stop trusting the easy story that “the model costs X” and start looking at the wrapper, the loop, the retries, the prompt churn, all the plumbing nobody likes to bill separately. That’s the real bill.
I think this piece points at something the agent ecosystem has been trying to gloss over: if you buy an agentic coding tool, you are not just buying model access. You are buying a harness with opinions about planning, verification, context retention, and when to keep talking to itself. Those opinions can be expensive. Sometimes they’re worth it. Sometimes they’re just noisy.
What I find more interesting than the benchmark headline is the implication for tool choice. People compare Claude Code, Aider, and whatever else as if the main question is “which model is best?” But if two tools can run the same model and still burn wildly different amounts of tokens, then the real decision is partly about software architecture and restraint. A tool that is a little less chatty, a little less eager to re-state the obvious, may be dramatically cheaper without being smarter in any meaningful sense.
I’d be cautious, though, about over-reading a benchmark like this. Identical model does not mean identical usefulness. A tighter harness might save tokens because it gives up earlier, asks less, or leans on simpler flows that work for the benchmark but frustrate you in real work. So the obvious next question is not “which one is cheapest?” but “what did the cheaper one leave out, and did I actually need it?” The article seems to nudge you toward that question, which I appreciate.
If I were evaluating these tools, I wouldn’t start with benchmark scores. I’d run my own ugly little repo through them and watch for three things: how often they reopen context they already had, how much they loop on the same failed idea, and whether they spend tokens to sound thoughtful instead of being helpful. That’s where agent costs usually hide.
Reference: Aider, Claude Code, and OpenClaw ran an identical model. Token use varied 70-fold.