Author

aiterms

What is top-p / nucleus sampling?

Top-p sampling, also called nucleus sampling, is a way to make a language model pick the next token from only the smallest set of likely options whose combined probability reaches a chosen threshold.

Why it matters

If you always choose the single most likely next token, model outputs can become repetitive and brittle. If you sample from all tokens, outputs can get noisy and incoherent.

Top-p gives you a middle ground: it keeps the model flexible, but trims away the long tail of very unlikely tokens before sampling. In practice, teams use it to control creativity and randomness without hard-coding a fixed number of candidates.

How it works

1. The model assigns a probability to every possible next token.
2. You sort tokens from most likely to least likely.
3. You keep adding tokens until their cumulative probability reaches p — for example, 0.9.
4. You discard the rest, renormalize the remaining probabilities, and sample one token from that smaller set.

The key idea is that the set size is dynamic. On a predictable prompt, the nucleus may be small. On a more open-ended prompt, it may be larger. That is why top-p is often preferred over fixed-k sampling when you want the model to adapt to context.

Top-p is usually paired with temperature. Temperature changes how peaked or flat the distribution is; top-p changes which tokens are even eligible to be sampled.

Tiny concrete example

Suppose the next-token probabilities are:

• A: 0.40
• B: 0.25
• C: 0.15
• D: 0.10
• E: 0.10

If top_p = 0.80, you keep A + B + C = 0.80 and drop D and E.

Then the model samples from just A, B, and C, after renormalizing their probabilities.

Common pitfalls / when NOT to use it

• Don’t treat p as a quality score. A higher p is not automatically “better”; it just allows a larger candidate set.
• Don’t assume it guarantees correctness. It only changes sampling behavior; it does not make the model more factual.
• Don’t over-tune both temperature and top-p blindly. If both are high, outputs can become overly random.
• Don’t use it when you need deterministic output. For exact reproducibility, use greedy decoding or fixed-seed sampling if your stack supports it.
• Don’t confuse it with top-k. Top-k keeps a fixed number of tokens; top-p keeps enough tokens to reach a probability threshold.

Related terms

• Top-k sampling
• Temperature
• Greedy decoding
• Beam search

Related terms

• What is a token (and tokenization)?
• What is a transformer?
• What is attention (self-attention)?
• What is temperature in an LLM?
• What is a mixture-of-experts (MoE) model?
• What is a parameter (model size)?
• What is the KV cache?
• What is a hallucination?
• What is grounding?

同じ著者の記事

What is temperature in an LLM?

Temperature is a sampling setting that controls how random or deterministic an LLM’s next-word choices are when it generates text. Temperature is one of the simplest ways to trade off consistency versus variety. Use a lower temperature when you want stable, repeatable answers: classification-like outputs, structured data, summaries, or code. Use a higher temperature when you want more diversity: brainstorming, creative writing, alternate phrasings, or exploratory chat. In practice, many

papoo.work

What is attention (self-attention)?

Attention, or self-attention, is a way for a model to decide which other words or tokens in the same input matter most when interpreting each token. Self-attention solves a basic language problem: the meaning of a word often depends on other words nearby or far away. It lets a model build a context-aware representation of each token instead of treating tokens mostly independently or relying only on fixed-size windows. In practice, this is why Transformers work so well for text, code, and other s

papoo.work

What is a transformer?

A transformer is a neural network architecture that processes sequences by using attention to compare each token with the others, making it a flexible foundation for modern language models and many other sequence tasks. Before transformers, many sequence models had to read text step by step, which made long-range dependencies harder to handle and training slower to parallelize. Transformers solve this by letting the model look at many positions in the input at once. In practice, you reach for a

papoo.work

What is a token (and tokenization)?

A token is a chunk of text that a language model reads and predicts, and tokenization is the process of splitting text into those chunks. LLMs do not operate directly on raw characters or words; they operate on tokens. That matters because: Input limits are measured in tokens, not characters. Cost and latency are often tied to token count in LLM APIs. Model quality depends on how text is tokenized: the same sentence can become very different token sequences across tokenizers.

papoo.work

What is structured output (JSON mode)?

Structured output, often called JSON mode, is a way to ask an LLM to return data in a fixed machine-readable format instead of free-form text. When you want an answer that another program can safely parse, plain chat text is fragile. A model might add extra words, change field names, or wrap the result in an explanation. Structured output helps when you need: predictable API responses reliable extraction into downstream systems cleaner automation for forms, workflows, or tools outputs that match

papoo.work

What is prompt engineering?

Prompt engineering is the practice of designing and refining the text you give to a language model so it produces more useful, reliable, and specific outputs. Large language models are sensitive to how you ask. A vague request can produce a vague answer; a well-structured prompt can improve format, scope, tone, and task performance. You reach for prompt engineering when you want to: get a model to follow instructions more consistently, steer it toward a role, style, or output format, reduce back

papoo.work

What is a system prompt?

A system prompt is the instruction text that sets the assistant’s role, behavior, and boundaries before the user’s message is processed. System prompts help you control how an LLM behaves without changing the model itself. They’re useful when you want consistent tone, safety rules, formatting, or task-specific behavior across many requests. In practice, teams use system prompts to: keep outputs on-brand, enforce policies or guardrails, define what the assistant should and should not do, reduce a

papoo.work

What is few-shot prompting?

Few-shot prompting, also called in-context learning, is a way of asking a language model to do a task by giving it a few example inputs and outputs in the prompt. Few-shot prompting helps when you want the model to follow a pattern, format, or classification rule without training or fine-tuning a new model. You’d reach for it when: the task is simple but the output needs to be consistent, you can show the model 1–5 good examples, you want to prototype quickly before investing in fine-tuning

papoo.work

What is zero-shot prompting?

Zero-shot prompting is asking an AI model to do a task without giving it any worked examples first. It is the simplest way to use a large language model: you describe the task in plain language and let the model infer the pattern from its pretraining. That makes it useful when you want: a fast baseline before writing better prompts simple classification, extraction, or drafting tasks lower prompt length and less example curation a test of whether the model already “gets” the task well enough In

papoo.work

What is chain-of-thought prompting?

Chain-of-thought prompting, often abbreviated CoT, is a way of asking a language model to produce intermediate reasoning steps before giving the final answer. It is used to improve performance on tasks that benefit from multi-step reasoning, such as arithmetic, logic, and some question-answering problems. In practice, teams reach for CoT when a direct one-line answer is more likely to be wrong than a stepwise solution. It can also make model behavior easier to inspect: if the model shows its

papoo.work