Author

aiterms

What is prompt caching?

Prompt caching is a way to reuse the cost and latency of processing the same prompt prefix instead of sending or recomputing it from scratch every time.

Why it matters

If your app sends a long, repeated system prompt, policy block, tool schema, or document context on many requests, you can end up paying to re-process the same text over and over. Prompt caching helps reduce that waste.

In practice, teams reach for it when:

• a large part of the prompt stays the same across requests,
• they want lower latency for repeated calls,
• they are building chat apps, copilots, or agent workflows with stable context.

It is especially useful when the “front half” of the prompt is fixed and only the user-specific part changes.

How it works

The exact mechanism depends on the model provider, but the basic idea is the same: if a new request starts with a prefix the system has already seen, that prefix can be reused from cache instead of being re-encoded or re-billed in full.

A common pattern is:

1. You send a prompt with a long stable prefix, such as instructions and reference text.
2. The model provider stores internal state for that prefix, keyed by the exact text and formatting.
3. On later requests with the same prefix, the provider can skip work for that shared part and only process the new suffix.

Important nuance: prompt caching is usually exact-match and prefix-based. Small changes in whitespace, ordering, or formatting can invalidate the cache. Also, caching is usually an implementation feature of the model service, not something the model itself “understands.”

Tiny concrete example

Suppose your app always sends this prefix:

System: You are a support assistant for Acme.
Use the product policy below.
[20 pages of policy text...]

Then each user request adds only the new question:

User: How do I reset my password?

If the prefix is cacheable and unchanged, the service may reuse the cached processing for the system/policy text and only handle the user question as new input.

Common pitfalls / when NOT to use it

• Your prompt changes too often. If the “fixed” part is actually different on most requests, caching won’t help much.
• You rely on exact formatting but keep changing it. Reordering sections, changing whitespace, or inserting dynamic timestamps can break cache hits.
• You assume it improves model quality. It is an efficiency feature, not a reasoning feature.
• You put highly dynamic content in the cached prefix. Keep the cacheable part as stable as possible.
• You expect all providers to behave the same. Prompt caching is provider-specific in details like cache lifetime, pricing effects, and what counts as a hit.

In short: use prompt caching when you have a large repeated prefix and want to save time or cost, but don’t treat it like a substitute for prompt design or context management.

Related terms

• What is prompt engineering?
• What is a system prompt?
• What is few-shot prompting?
• What is zero-shot prompting?
• What is chain-of-thought prompting?
• What is context engineering?
• What is a context window?
• What is structured output (JSON mode)?
• What is prompt injection?

同じ著者の記事

What is a context window?

A context window is the amount of text a language model can read and keep in mind at once while it generates a response. The context window sets the practical limit for what you can ask a model to consider in a single turn: your prompt, system instructions, chat history, retrieved documents, and the model’s own recent output all compete for that space. In practice, it matters when you want the model to: follow long instructions, answer questions about a large document, maintain conversation stat

papoo.work

What is context engineering?

Context engineering is the practice of deliberately assembling the right information, instructions, tools, and examples into an AI model’s context so it can do a task reliably. Large language models do not “know” what you want unless you provide it in the prompt, surrounding messages, retrieved documents, tool outputs, or examples. Context engineering helps you get better accuracy, fewer hallucinations, and more consistent behavior by shaping what the model sees before it answers. In practice, i

papoo.work

What is chain-of-thought prompting?

Chain-of-thought prompting, often abbreviated CoT, is a way of asking a language model to produce intermediate reasoning steps before giving the final answer. It is used to improve performance on tasks that benefit from multi-step reasoning, such as arithmetic, logic, and some question-answering problems. In practice, teams reach for CoT when a direct one-line answer is more likely to be wrong than a stepwise solution. It can also make model behavior easier to inspect: if the model shows its

papoo.work

What is zero-shot prompting?

Zero-shot prompting is asking an AI model to do a task without giving it any worked examples first. It is the simplest way to use a large language model: you describe the task in plain language and let the model infer the pattern from its pretraining. That makes it useful when you want: a fast baseline before writing better prompts simple classification, extraction, or drafting tasks lower prompt length and less example curation a test of whether the model already “gets” the task well enough In

papoo.work

What is few-shot prompting?

Few-shot prompting, also called in-context learning, is a way of asking a language model to do a task by giving it a few example inputs and outputs in the prompt. Few-shot prompting helps when you want the model to follow a pattern, format, or classification rule without training or fine-tuning a new model. You’d reach for it when: the task is simple but the output needs to be consistent, you can show the model 1–5 good examples, you want to prototype quickly before investing in fine-tuning

papoo.work

What is a system prompt?

A system prompt is the instruction text that sets the assistant’s role, behavior, and boundaries before the user’s message is processed. System prompts help you control how an LLM behaves without changing the model itself. They’re useful when you want consistent tone, safety rules, formatting, or task-specific behavior across many requests. In practice, teams use system prompts to: keep outputs on-brand, enforce policies or guardrails, define what the assistant should and should not do, reduce a

papoo.work

What is prompt engineering?

Prompt engineering is the practice of designing and refining the text you give to a language model so it produces more useful, reliable, and specific outputs. Large language models are sensitive to how you ask. A vague request can produce a vague answer; a well-structured prompt can improve format, scope, tone, and task performance. You reach for prompt engineering when you want to: get a model to follow instructions more consistently, steer it toward a role, style, or output format, reduce back

papoo.work

What is structured output (JSON mode)?

Structured output, often called JSON mode, is a way to ask an LLM to return data in a fixed machine-readable format instead of free-form text. When you want an answer that another program can safely parse, plain chat text is fragile. A model might add extra words, change field names, or wrap the result in an explanation. Structured output helps when you need: predictable API responses reliable extraction into downstream systems cleaner automation for forms, workflows, or tools outputs that match

papoo.work

What is contextual / late chunking?

Contextual chunking, also called late chunking, is a way to split a document into retrieval chunks after a model has already encoded the whole document, so each chunk keeps some awareness of the surrounding context. Normal chunking cuts text first and then embeds each piece alone. That is simple, but it can make chunks semantically thin: a paragraph may be ambiguous without the surrounding sections, headings, or earlier definitions. Late chunking helps when you want better retrieval quality

papoo.work

What is a knowledge base for RAG?

A knowledge base for RAG is the collection of documents or records a system retrieves from to ground an LLM’s answer in relevant source material. RAG, short for retrieval-augmented generation, works best when the model can search a useful, trustworthy set of content before it writes. That content is the knowledge base. You’d reach for one when the model needs to answer questions from: company docs support articles policies and handbooks product specs research notes tickets or tickets-like record

papoo.work