What jumped out at me here is not that prompt caching can be expensive when you misuse it. It’s that it can look like it’s working while quietly being almost useless. A 78% hit rate sounds decent until you realize the shape of the remaining misses can make the whole thing much pricier than if you had left caching off. That’s the part I’d actually want to know in my own logs: not “is caching enabled?” but “what is the prefix, byte for byte, and who changed it?”
The timestamp bug is the obvious cautionary tale, but the more interesting one is the Python set ordering issue. That’s the kind of bug that slips through because local testing lies to you. One process, one worker, one stable order, everything looks fine. Then production adds parallelism and the cache starts behaving like a slot machine. I think this is a good reminder that anything used to build a cached prompt needs to be treated like part of the public API. If it can vary across workers, it can vary enough to cost money.
I also like that the author didn’t stop at “remove the timestamp.” Moving volatile state into the latest user message is the right instinct, because it separates stable prefix from session-specific noise. That said, I’m slightly wary of the implied neatness of the fix. There’s still a deeper question about how much conversation history they’re willing to keep re-sending uncached. They mention that as the next experiment, and that’s probably where the real cost lives once the prefix is under control.
The biggest practical takeaway for me is not the specific bug, but the monitoring habit. cache_read_input_tokens and cache_creation_input_tokens should probably be first-class metrics in any Claude app that relies on prompt caching. I’d go further and hash the constructed prefix in CI, exactly because runtime behavior can be too slippery to trust. If you’re using caching and not inspecting the usage fields, you’re basically flying blind.
Reference: Claude Prompt Caching Hit Rate Was 0%. One Timestamp Did It
