PaPoo
cover

Claude’s vocabulary problem feels real, but I’m not fully convinced by the framing

What immediately caught my eye is the premise itself: that there’s a “load-bearing vocabulary” in Claude. That sounds trivial on first read, but I think it points at something practical that people building with these models run into all the time: some prompts work not because they’re conceptually special, but because the model has learned to treat certain words as signals of seriousness, structure, or intent. The annoying part is that this can be true even when nobody involved can explain exactly why.

What I like about that idea is that it matches the lived experience of prompt engineering better than the glossy narratives do. You change a single word, and the model starts behaving differently. Not in a mystical way, just in the way large language models latch onto patterns from training data. “Load-bearing vocabulary” is a useful phrase for that. It suggests that some prompt words are doing hidden work, and if you remove them, the whole thing gets wobbly.

That said, I’m always suspicious when people make this sound more deterministic than it is. I’d want to know whether the observed effect is genuinely Claude-specific, or just a general property of any sufficiently large model. I also wonder how much of this is model behavior and how much is user selection bias. We remember the prompts that break when we edit them. We forget the dozens that don’t. If the article is arguing that Claude has a distinctive vocabulary sensitivity, I think the burden is on the evidence.

Still, as a builder, I’d take the experiment seriously. If I were testing this, I’d try the same task with controlled prompt rewrites: keep semantics as close as possible, swap in plain language for “magic” words, and see where performance drops. Not because I expect a clean answer, but because this is exactly the kind of thing that can quietly affect agent reliability. A prompt that works in a demo and then falls apart in production often isn’t “bad logic”; it’s often missing some brittle cue the model had been leaning on.

The phrase itself is useful because it pushes against a lazy assumption: that natural language instructions are fully semantic. They aren’t. In practice, they’re part semantics, part affordance, part training-data superstition. Claude, like the other frontier models, is probably more sensitive to that mix than many users want to admit.


Reference: The load-bearing vocabulary of Claude

同じ著者の記事