Author

aiterms

What is temperature in an LLM?

Temperature is a sampling setting that controls how random or deterministic an LLM’s next-word choices are when it generates text.

Why it matters

Temperature is one of the simplest ways to trade off consistency versus variety.

• Use a lower temperature when you want stable, repeatable answers: classification-like outputs, structured data, summaries, or code.
• Use a higher temperature when you want more diversity: brainstorming, creative writing, alternate phrasings, or exploratory chat.

In practice, many teams start with a low temperature for production workflows and raise it only when they explicitly want more variation.

How it works

An LLM produces a probability distribution over possible next tokens. Temperature changes how sharply or evenly that distribution is sampled.

• Lower temperature makes the model prefer the most likely tokens more strongly.
• Higher temperature flattens the distribution, giving lower-probability tokens more chance to appear.
• At temperature = 0, many systems use the most likely token at each step, which is effectively greedy decoding. This tends to be the most deterministic setting.
• A very high temperature can make outputs less coherent, because the model is choosing from much less likely continuations.

A useful mental model: temperature does not change what the model “knows”; it changes how adventurous the generator is when choosing among candidate continuations.

Tiny concrete example

Suppose the model is considering the next word after:

“The best way to explain temperature is…”

At low temperature, it might repeatedly pick the most likely continuation, such as:

“to think of it as a randomness control.”

At higher temperature, it may more often choose alternate phrasings, such as:

“to treat it as a knob for variety.”

“to view it as a sampling dial.”

Same underlying model, different sampling behavior.

Common pitfalls / when NOT to use it

• Temperature is not “creativity” by itself. It only changes sampling randomness; prompt quality still matters a lot.
• It does not guarantee factuality. Lower temperature can make answers more consistent, but not necessarily more correct.
• It is not a universal standard across APIs. Different vendors may expose temperature with slightly different sampling stacks or defaults.
• Don’t rely on temperature for strict determinism in all cases. If you need exact repeatability, also watch for other sources of variation like model version, system prompts, tool outputs, and nondeterministic infrastructure.
• Don’t crank it up for tasks with rigid output formats. High temperature can break JSON, schemas, or exact phrasing requirements.

In short: if you want steadier outputs, lower it; if you want more variety, raise it.

Related terms

• What is a token (and tokenization)?
• What is a transformer?
• What is attention (self-attention)?
• What is top-p / nucleus sampling?
• What is a mixture-of-experts (MoE) model?
• What is a parameter (model size)?
• What is the KV cache?
• What is a hallucination?
• What is grounding?

同じ著者の記事

What is top-p / nucleus sampling?

Top-p sampling, also called nucleus sampling, is a way to make a language model pick the next token from only the smallest set of likely options whose combined probability reaches a chosen threshold. If you always choose the single most likely next token, model outputs can become repetitive and brittle. If you sample from all tokens, outputs can get noisy and incoherent. Top-p gives you a middle ground: it keeps the model flexible, but trims away the long tail of very unlikely tokens before samp

papoo.work

What is attention (self-attention)?

Attention, or self-attention, is a way for a model to decide which other words or tokens in the same input matter most when interpreting each token. Self-attention solves a basic language problem: the meaning of a word often depends on other words nearby or far away. It lets a model build a context-aware representation of each token instead of treating tokens mostly independently or relying only on fixed-size windows. In practice, this is why Transformers work so well for text, code, and other s

papoo.work

What is a transformer?

A transformer is a neural network architecture that processes sequences by using attention to compare each token with the others, making it a flexible foundation for modern language models and many other sequence tasks. Before transformers, many sequence models had to read text step by step, which made long-range dependencies harder to handle and training slower to parallelize. Transformers solve this by letting the model look at many positions in the input at once. In practice, you reach for a

papoo.work

What is a token (and tokenization)?

A token is a chunk of text that a language model reads and predicts, and tokenization is the process of splitting text into those chunks. LLMs do not operate directly on raw characters or words; they operate on tokens. That matters because: Input limits are measured in tokens, not characters. Cost and latency are often tied to token count in LLM APIs. Model quality depends on how text is tokenized: the same sentence can become very different token sequences across tokenizers.

papoo.work

What is structured output (JSON mode)?

Structured output, often called JSON mode, is a way to ask an LLM to return data in a fixed machine-readable format instead of free-form text. When you want an answer that another program can safely parse, plain chat text is fragile. A model might add extra words, change field names, or wrap the result in an explanation. Structured output helps when you need: predictable API responses reliable extraction into downstream systems cleaner automation for forms, workflows, or tools outputs that match

papoo.work

What is prompt engineering?

Prompt engineering is the practice of designing and refining the text you give to a language model so it produces more useful, reliable, and specific outputs. Large language models are sensitive to how you ask. A vague request can produce a vague answer; a well-structured prompt can improve format, scope, tone, and task performance. You reach for prompt engineering when you want to: get a model to follow instructions more consistently, steer it toward a role, style, or output format, reduce back

papoo.work

What is a system prompt?

A system prompt is the instruction text that sets the assistant’s role, behavior, and boundaries before the user’s message is processed. System prompts help you control how an LLM behaves without changing the model itself. They’re useful when you want consistent tone, safety rules, formatting, or task-specific behavior across many requests. In practice, teams use system prompts to: keep outputs on-brand, enforce policies or guardrails, define what the assistant should and should not do, reduce a

papoo.work

What is few-shot prompting?

Few-shot prompting, also called in-context learning, is a way of asking a language model to do a task by giving it a few example inputs and outputs in the prompt. Few-shot prompting helps when you want the model to follow a pattern, format, or classification rule without training or fine-tuning a new model. You’d reach for it when: the task is simple but the output needs to be consistent, you can show the model 1–5 good examples, you want to prototype quickly before investing in fine-tuning

papoo.work

What is zero-shot prompting?

Zero-shot prompting is asking an AI model to do a task without giving it any worked examples first. It is the simplest way to use a large language model: you describe the task in plain language and let the model infer the pattern from its pretraining. That makes it useful when you want: a fast baseline before writing better prompts simple classification, extraction, or drafting tasks lower prompt length and less example curation a test of whether the model already “gets” the task well enough In

papoo.work

What is chain-of-thought prompting?

Chain-of-thought prompting, often abbreviated CoT, is a way of asking a language model to produce intermediate reasoning steps before giving the final answer. It is used to improve performance on tasks that benefit from multi-step reasoning, such as arithmetic, logic, and some question-answering problems. In practice, teams reach for CoT when a direct one-line answer is more likely to be wrong than a stepwise solution. It can also make model behavior easier to inspect: if the model shows its

papoo.work