Author

aiterms

What is a token (and tokenization)?

A token is a chunk of text that a language model reads and predicts, and tokenization is the process of splitting text into those chunks.

Why it matters

LLMs do not operate directly on raw characters or words; they operate on tokens. That matters because:

• Input limits are measured in tokens, not characters.
• Cost and latency are often tied to token count in LLM APIs.
• Model quality depends on how text is tokenized: the same sentence can become very different token sequences across tokenizers.
• “Prompt length” and “context window” are token-based concepts, so tokenization affects how much you can fit into a prompt or conversation.

In practice, if you are building with LLMs, tokenization is one of the first things you should check before debugging “why did this get truncated?” or “why was this so expensive?”

How it works

Most modern tokenizers do not split text only by spaces. They usually break text into subword units using a learned vocabulary. This is a practical compromise:

1. Common pieces stay intact. Frequent words or fragments may become single tokens.
2. Rare or new words get split. A long technical term, name, or misspelling may become several tokens.
3. The model maps tokens to numbers. Each token is assigned an integer ID, which the model can process.

A token can be:

• a whole word,
• part of a word,
• punctuation,
• whitespace,
• or a byte-level fragment, depending on the tokenizer.

This is why “one word = one token” is only sometimes true. In many tokenizers, "unbelievable" may be one token or several pieces, while "hello" and "!" might each be separate tokens. Exact behavior depends on the tokenizer used by the model.

Tiny concrete example

Text:

I love tokenization!

A tokenizer might split it roughly like this:

• I
• love
• token
• ization
• !

Another tokenizer might split it differently, for example:

• I
• love
• tokenization
• !

Same sentence, different tokenization. The model then works with token IDs rather than the original text.

Common pitfalls / when NOT to use it

• Do not assume token counts from characters or words. English, code, emoji, and non-Latin scripts can tokenize very differently.
• Do not assume tokenization is universal. Different model families use different tokenizers, so the same prompt may cost different amounts or fit differently.
• Do not use token counts as a proxy for meaning. Tokens are a modeling convenience, not linguistic units.
• Do not hand-encode text unless you have to. Most app developers should rely on the model’s tokenizer or API tooling.
• Be careful with code, JSON, and multilingual text. These often produce more tokens than people expect.

If you need exact behavior, use the tokenizer associated with the specific model you plan to call.

Related terms

• What is a transformer?
• What is attention (self-attention)?
• What is temperature in an LLM?
• What is top-p / nucleus sampling?
• What is a mixture-of-experts (MoE) model?
• What is a parameter (model size)?
• What is the KV cache?
• What is a hallucination?
• What is grounding?

同じ著者の記事

What is top-p / nucleus sampling?

Top-p sampling, also called nucleus sampling, is a way to make a language model pick the next token from only the smallest set of likely options whose combined probability reaches a chosen threshold. If you always choose the single most likely next token, model outputs can become repetitive and brittle. If you sample from all tokens, outputs can get noisy and incoherent. Top-p gives you a middle ground: it keeps the model flexible, but trims away the long tail of very unlikely tokens before samp

papoo.work

What is temperature in an LLM?

Temperature is a sampling setting that controls how random or deterministic an LLM’s next-word choices are when it generates text. Temperature is one of the simplest ways to trade off consistency versus variety. Use a lower temperature when you want stable, repeatable answers: classification-like outputs, structured data, summaries, or code. Use a higher temperature when you want more diversity: brainstorming, creative writing, alternate phrasings, or exploratory chat. In practice, many

papoo.work

What is attention (self-attention)?

Attention, or self-attention, is a way for a model to decide which other words or tokens in the same input matter most when interpreting each token. Self-attention solves a basic language problem: the meaning of a word often depends on other words nearby or far away. It lets a model build a context-aware representation of each token instead of treating tokens mostly independently or relying only on fixed-size windows. In practice, this is why Transformers work so well for text, code, and other s

papoo.work

What is a transformer?

A transformer is a neural network architecture that processes sequences by using attention to compare each token with the others, making it a flexible foundation for modern language models and many other sequence tasks. Before transformers, many sequence models had to read text step by step, which made long-range dependencies harder to handle and training slower to parallelize. Transformers solve this by letting the model look at many positions in the input at once. In practice, you reach for a

papoo.work

What is structured output (JSON mode)?

Structured output, often called JSON mode, is a way to ask an LLM to return data in a fixed machine-readable format instead of free-form text. When you want an answer that another program can safely parse, plain chat text is fragile. A model might add extra words, change field names, or wrap the result in an explanation. Structured output helps when you need: predictable API responses reliable extraction into downstream systems cleaner automation for forms, workflows, or tools outputs that match

papoo.work

What is prompt engineering?

Prompt engineering is the practice of designing and refining the text you give to a language model so it produces more useful, reliable, and specific outputs. Large language models are sensitive to how you ask. A vague request can produce a vague answer; a well-structured prompt can improve format, scope, tone, and task performance. You reach for prompt engineering when you want to: get a model to follow instructions more consistently, steer it toward a role, style, or output format, reduce back

papoo.work

What is a system prompt?

A system prompt is the instruction text that sets the assistant’s role, behavior, and boundaries before the user’s message is processed. System prompts help you control how an LLM behaves without changing the model itself. They’re useful when you want consistent tone, safety rules, formatting, or task-specific behavior across many requests. In practice, teams use system prompts to: keep outputs on-brand, enforce policies or guardrails, define what the assistant should and should not do, reduce a

papoo.work

What is few-shot prompting?

Few-shot prompting, also called in-context learning, is a way of asking a language model to do a task by giving it a few example inputs and outputs in the prompt. Few-shot prompting helps when you want the model to follow a pattern, format, or classification rule without training or fine-tuning a new model. You’d reach for it when: the task is simple but the output needs to be consistent, you can show the model 1–5 good examples, you want to prototype quickly before investing in fine-tuning

papoo.work

What is zero-shot prompting?

Zero-shot prompting is asking an AI model to do a task without giving it any worked examples first. It is the simplest way to use a large language model: you describe the task in plain language and let the model infer the pattern from its pretraining. That makes it useful when you want: a fast baseline before writing better prompts simple classification, extraction, or drafting tasks lower prompt length and less example curation a test of whether the model already “gets” the task well enough In

papoo.work

What is chain-of-thought prompting?

Chain-of-thought prompting, often abbreviated CoT, is a way of asking a language model to produce intermediate reasoning steps before giving the final answer. It is used to improve performance on tasks that benefit from multi-step reasoning, such as arithmetic, logic, and some question-answering problems. In practice, teams reach for CoT when a direct one-line answer is more likely to be wrong than a stepwise solution. It can also make model behavior easier to inspect: if the model shows its

papoo.work