What jumped out at me is how aggressively Armature is trying to define a new layer of the stack: not app analytics, not LLM observability, but “what users do” once the UI disappears and Claude, ChatGPT, or some other client becomes the actual front door. That’s a real problem, and the pitch makes sense on first read. But it also has a whiff of category invention, which means it could be either prescient or a very polished answer in search of a problem that only some teams feel.
The part I believe immediately is the analytics angle. If your MCP server is being hit through a client you don’t control, the old PostHog/Amplitude/Mixpanel model stops being very useful. You can’t infer intent from clicks because there aren’t any. You’re left with tool calls, retries, loops, and whatever the agent decided to say to itself. Armature seems to be saying: we can rebuild those sessions, infer the user’s intent, and surface the failures that matter even when the API technically returned 200 OK. That last bit is the strongest claim on the page, and probably the most valuable one if it works.
The evals story is where I’m a little more skeptical, or at least cautious. “Turn the workflows your users actually run into evals” is a nice line, but there’s a gap between real production traces and a good eval suite. The source suggests Armature can draft evals from analytics, or let you write them from scratch, and then run real agents across models and harnesses with a judge scoring the result. Fine. But the hard part is not generating a test case template. It’s deciding what counts as success in a workflow that may be messy, policy-heavy, and highly context dependent. I’d want to see how much human tuning is actually needed after the initial draft.
I also think the product positioning is clever in a slightly slippery way. They’re not just saying “we observe MCP servers.” They’re saying they observe “Claude Connector, ChatGPT App or MCP,” and they support “any client your users bring.” That’s broad enough to sound future-proof, but it also raises questions about how much is truly standardized versus how much is just wrapped in a nice onboarding story. The FAQ says the SDK sits in your MCP server backend and doesn’t change behavior for users. If that’s true, great. If not, this kind of instrumentation can get messy fast.
The one thing that feels genuinely timely is the emphasis on sessions as product evidence. In agent land, the most useful signal may not be a feature request or a conversion funnel. It may be a trace that shows the model looping on auth scope, misunderstanding “refund,” or truncating a report because pagination got in the way. That’s concrete. That’s the kind of thing a product team can fix. If Armature can reliably surface that without burying teams in noisy traces, it has something real.
I’d still want to know how often the “user intent” and “agent thinking” layers are inferred versus actually captured, because that distinction matters. If the system is mostly reconstructing intent from tool calls and text, the confidence level should be treated carefully. If it’s just using a labeler to make traces readable, that’s useful. If it’s pretending to know more than it does, that’s a problem.
What I’d try first is the replay view, not the marketing claims. Show me a broken workflow, the inferred intent, the loop, and the exact spot where the agent went sideways. If that feels better than digging through raw logs, then the product is onto something. If not, then this is another observability tool with a nicer story.
Reference: Armature · Analytics and evals for your MCP
