What jumped out at me wasn’t the “18% better” line. It was the amount of engineering they had to build just to make that claim feel remotely credible. That’s the part most companies hand-wave away. Here, they at least seem to understand that agent benchmarks are brittle as hell unless you control the data, the prompt, the tools, and the grading.
I’m still not fully convinced by any benchmark that ends with an LLM judge, though. Blind grading sounds cleaner than a human eyeballing outputs, but it’s still an LLM deciding whether an investigation was “right enough.” That might be fine for relative comparisons, especially if they keep the judge blind to implementation details, but it’s not the same as proving real-world incident response quality. The article seems aware of that, which helps. It doesn’t pretend the setup is perfect.
What I do like is the synthetic telemetry angle. Using public observability datasets would have been an easy shortcut, and probably a bad one. If a model has seen a dataset or something close to it during training, your benchmark turns into memory recall theater. Their answer is to generate incidents from scratch, seed them deterministically, and bury the real signal under realistic noise and distractors. That’s much closer to how production failures actually feel: annoying, ambiguous, and full of traps for the first hypothesis that looks good.
The most convincing detail is probably the one about scale. If the root cause is hidden among tens of millions of spans and logs, you’re no longer testing whether the agent can spot an error string. You’re testing whether it can avoid being tricked by the obvious thing. That’s a real difference. A lot of “AI for ops” demos collapse into a fancy grep with confidence. These scenarios at least seem designed to punish that.
I also think the ClickStack MCP vs raw SQL distinction is the real product argument here. Raw SQL is more flexible, but flexibility is not free when the agent has to invent its own investigation workflow. Giving it higher-level primitives like pattern finding, time-window comparison, and cross-signal movement is basically admitting that LLMs are better at choosing among tools than composing every query from first principles. That feels right to me. The cleverness is in deciding how much abstraction to provide before you start boxing the model in.
Still, there’s an implicit bet here: that the abstractions are general enough to survive real incidents, not just the five scenarios they designed. Maybe they are. Maybe the MCP really is a cleaner interface for SRE agents. But I’d want to see how often the “better” tool becomes the wrong tool when the failure mode doesn’t look like the benchmark. That’s the missing test. Production rarely gives you a neat Simpson’s Paradox puzzle on demand.
What I’d try next, if I were using this setup, is not just comparing tool interfaces but measuring when the agent chooses the wrong investigative path even with the right tools available. That seems more revealing than a single score. The article makes a good case that they’ve built a serious harness. Whether it generalizes outside ClickHouse’s carefully staged incidents is still the question that matters.
Reference: How we build and evaluate our MCP server for SRE agents | ClickHouse
