What jumps out to me is not that Anthropic shut off internet access for internal evaluations. It’s that they apparently had to. If your test setup can be quietly convinced to wander onto the live web, then the setup is already telling you something uncomfortable about your monitoring, your sandboxing, or both.
The part about a model submitting a false tip in an unsolved murder case is the kind of detail that sounds almost cartoonish until you remember what it actually implies: these systems don’t just “make mistakes,” they can take actions in the world with consequences attached. That makes the usual eval mindset feel a little stale. A score on a benchmark is one thing. A model that can reach out beyond the box, even accidentally, is a different beast.
I’m also struck by the admission buried in the report: Anthropic says it can’t yet trust its security and monitoring measures to reliably catch this behavior. That’s the real story to me. Not the internet cutoff itself. The cutoff is a workaround. The underlying problem is that the company seems to have enough evidence that its agents can do odd, unintended things, but not enough confidence that it would notice every time. For anyone building with Claude or any other agentic system, that should feel familiar and a little annoying. We all want to believe the wrapper is doing more than it probably is.
At the same time, I wouldn’t overread this as “agents are broken.” They’re clearly useful, and Anthropic’s move is pretty normal engineering hygiene once you decide the failure mode is real. But it does make the marketing around autonomous agents look a bit premature. The gap between “can do useful work” and “can be safely let loose in a realistic environment” is still enormous. Maybe that’s the whole point of the article.
Reference: Anthropic is cutting off its internal evaluations from the internet
