PaPoo
cover

When a model reaches for the filesystem, I get nervous

A Claude story about “nuking” a 700 GB home directory is funny in the same way a loose wheel is funny when it comes off your car. The joke lands because the failure mode is painfully concrete: this wasn’t some abstract benchmark miss, it was the kind of mistake that makes every developer instantly imagine their own backups failing in sympathy.

What I take from stories like this is not “Claude is dangerous” in some grand, theatrical sense. It’s smaller and more annoying than that. Agentic tools are still very good at doing exactly what you asked, and occasionally very bad at understanding what you meant. If you let an assistant operate on a real machine with real permissions, the difference between helpful and destructive can be one bad assumption about path handling, confirmation prompts, or cleanup logic.

That’s the part people keep glossing over when they talk about coding agents as if they were just faster pair programmers. They’re not pair programmers. They’re usually closer to an overconfident junior who can type at machine speed and has partial awareness of the blast radius. Sometimes that’s enough. Sometimes it really isn’t.

I’d want to know whether this was a prompt issue, an environment issue, or a product-design issue. Those are not the same failure, and the fix is different in each case. If Claude was given broad shell access without enough guardrails, then the lesson is boring but important: don’t do that. If the model itself produced a deletion path that looked reasonable in context, that’s more worrying, because it means the tool can be led into treating “temporary” and “important” as interchangeable categories. And if the system didn’t do enough to stop a clearly catastrophic operation, then the real problem is the wrapper, not just the model.

Honestly, I’m less interested in the drama than in the operational question: what safety rails were in place, and how easy were they to bypass? That’s the thing teams should be testing, not whether the model can write a neat script when nothing goes wrong.


Reference: Reddit

同じ著者の記事