Author

aiterms

What is attention (self-attention)?

Attention, or self-attention, is a way for a model to decide which other words or tokens in the same input matter most when interpreting each token.

Why it matters

Self-attention solves a basic language problem: the meaning of a word often depends on other words nearby or far away. It lets a model build a context-aware representation of each token instead of treating tokens mostly independently or relying only on fixed-size windows.

In practice, this is why Transformers work so well for text, code, and other sequence data. If you need a model to connect “it” to the right noun, or to link a later instruction to an earlier constraint, self-attention is the mechanism doing that work.

How it works

At a high level, each token produces three vectors:

• Query: what this token is looking for
• Key: what this token offers
• Value: the information to pass along

For a given token, the model compares its query to the keys of all tokens in the sequence. Those comparisons become weights: tokens that seem more relevant get higher weight. The model then takes a weighted sum of the corresponding values to produce a new representation for that token.

Because every token can attend to every other token, self-attention is good at modeling long-range dependencies. In Transformers, this is usually done in parallel across many tokens, and often with multi-head attention, where several attention “heads” learn different kinds of relationships at once.

A key point: self-attention is not a single rule for “importance.” It is a learned mechanism. The model figures out, during training, which patterns of attention help it do the task.

Tiny concrete example

Sentence:
“The trophy didn’t fit in the suitcase because it was too big.”

When processing “it”, a self-attention layer can put more weight on “trophy” than on “suitcase” if that helps the model infer that “it” refers to the trophy. In another sentence, the weights could shift to a different word depending on context.

Common pitfalls / when NOT to use it

• It is not human-like reasoning. Attention weights show what the model focused on, but they do not by themselves explain the model’s full decision.
• It is not the same as “memory.” Self-attention helps relate tokens inside the current context; it does not automatically create persistent long-term memory across sessions.
• It can be expensive on long inputs. Naive self-attention scales poorly with sequence length, so very long documents may need optimized variants or chunking.
• Don’t assume a high attention weight means causation. Attention is useful, but it is not a guaranteed explanation of why the output happened.

In practice, if you need a model to connect parts of the same input flexibly, self-attention is the right idea. If you need durable state across interactions, look elsewhere.

What is attention (self-attention)?

Attention, or self-attention, is a way for a model to decide which other words or tokens in the same input matter most when interpreting each token.

Why it matters

Self-attention solves a basic language problem: the meaning of a word often depends on other words nearby or far away. It lets a model build a context-aware representation of each token instead of treating tokens mostly independently or relying only on fixed-size windows.

In practice, this is why Transformers work so well for text, code, and other sequence data. If you need a model to connect “it” to the right noun, or to link a later instruction to an earlier constraint, self-attention is the mechanism doing that work.

How it works

At a high level, each token produces three vectors:

• Query: what this token is looking for
• Key: what this token offers
• Value: the information to pass along

For a given token, the model compares its query to the keys of all tokens in the sequence. Those comparisons become weights: tokens that seem more relevant get higher weight. The model then takes a weighted sum of the corresponding values to produce a new representation for that token.

Because every token can attend to every other token, self-attention is good at modeling long-range dependencies. In Transformers, this is usually done in parallel across many tokens, and often with multi-head attention, where several attention “heads” learn different kinds of relationships at once.

A key point: self-attention is not a single rule for “importance.” It is a learned mechanism. The model figures out, during training, which patterns of attention help it do the task.

Tiny concrete example

Sentence:
“The trophy didn’t fit in the suitcase because it was too big.”

When processing “it”, a self-attention layer can put more weight on “trophy” than on “suitcase” if that helps the model infer that “it” refers to the trophy. In another sentence, the weights could shift to a different word depending on context.

Common pitfalls / when NOT to use it

• It is not human-like reasoning. Attention weights show what the model focused on, but they do not by themselves explain the model’s full decision.
• It is not the same as “memory.” Self-attention helps relate tokens inside the current context; it does not automatically create persistent long-term memory across sessions.
• It can be expensive on long inputs. Naive self-attention scales poorly with sequence length, so very long documents may need optimized variants or chunking.
• Don’t assume a high attention weight means causation. Attention is useful, but it is not a guaranteed explanation of why the output happened.

In practice, if you need a model to connect parts of the same input flexibly, self-attention is the right idea. If you need durable state across interactions, look elsewhere.

Related terms

• What is a token (and tokenization)?
• What is a transformer?
• What is temperature in an LLM?
• What is top-p / nucleus sampling?
• What is a mixture-of-experts (MoE) model?
• What is a parameter (model size)?
• What is the KV cache?
• What is a hallucination?
• What is grounding?

同じ著者の記事

What is top-p / nucleus sampling?

Top-p sampling, also called nucleus sampling, is a way to make a language model pick the next token from only the smallest set of likely options whose combined probability reaches a chosen threshold. If you always choose the single most likely next token, model outputs can become repetitive and brittle. If you sample from all tokens, outputs can get noisy and incoherent. Top-p gives you a middle ground: it keeps the model flexible, but trims away the long tail of very unlikely tokens before samp

papoo.work

What is temperature in an LLM?

Temperature is a sampling setting that controls how random or deterministic an LLM’s next-word choices are when it generates text. Temperature is one of the simplest ways to trade off consistency versus variety. Use a lower temperature when you want stable, repeatable answers: classification-like outputs, structured data, summaries, or code. Use a higher temperature when you want more diversity: brainstorming, creative writing, alternate phrasings, or exploratory chat. In practice, many

papoo.work

What is a transformer?

A transformer is a neural network architecture that processes sequences by using attention to compare each token with the others, making it a flexible foundation for modern language models and many other sequence tasks. Before transformers, many sequence models had to read text step by step, which made long-range dependencies harder to handle and training slower to parallelize. Transformers solve this by letting the model look at many positions in the input at once. In practice, you reach for a

papoo.work

What is a token (and tokenization)?

A token is a chunk of text that a language model reads and predicts, and tokenization is the process of splitting text into those chunks. LLMs do not operate directly on raw characters or words; they operate on tokens. That matters because: Input limits are measured in tokens, not characters. Cost and latency are often tied to token count in LLM APIs. Model quality depends on how text is tokenized: the same sentence can become very different token sequences across tokenizers.

papoo.work

What is structured output (JSON mode)?

Structured output, often called JSON mode, is a way to ask an LLM to return data in a fixed machine-readable format instead of free-form text. When you want an answer that another program can safely parse, plain chat text is fragile. A model might add extra words, change field names, or wrap the result in an explanation. Structured output helps when you need: predictable API responses reliable extraction into downstream systems cleaner automation for forms, workflows, or tools outputs that match

papoo.work

What is prompt engineering?

Prompt engineering is the practice of designing and refining the text you give to a language model so it produces more useful, reliable, and specific outputs. Large language models are sensitive to how you ask. A vague request can produce a vague answer; a well-structured prompt can improve format, scope, tone, and task performance. You reach for prompt engineering when you want to: get a model to follow instructions more consistently, steer it toward a role, style, or output format, reduce back

papoo.work

What is a system prompt?

A system prompt is the instruction text that sets the assistant’s role, behavior, and boundaries before the user’s message is processed. System prompts help you control how an LLM behaves without changing the model itself. They’re useful when you want consistent tone, safety rules, formatting, or task-specific behavior across many requests. In practice, teams use system prompts to: keep outputs on-brand, enforce policies or guardrails, define what the assistant should and should not do, reduce a

papoo.work

What is few-shot prompting?

Few-shot prompting, also called in-context learning, is a way of asking a language model to do a task by giving it a few example inputs and outputs in the prompt. Few-shot prompting helps when you want the model to follow a pattern, format, or classification rule without training or fine-tuning a new model. You’d reach for it when: the task is simple but the output needs to be consistent, you can show the model 1–5 good examples, you want to prototype quickly before investing in fine-tuning

papoo.work

What is zero-shot prompting?

Zero-shot prompting is asking an AI model to do a task without giving it any worked examples first. It is the simplest way to use a large language model: you describe the task in plain language and let the model infer the pattern from its pretraining. That makes it useful when you want: a fast baseline before writing better prompts simple classification, extraction, or drafting tasks lower prompt length and less example curation a test of whether the model already “gets” the task well enough In

papoo.work

What is chain-of-thought prompting?

Chain-of-thought prompting, often abbreviated CoT, is a way of asking a language model to produce intermediate reasoning steps before giving the final answer. It is used to improve performance on tasks that benefit from multi-step reasoning, such as arithmetic, logic, and some question-answering problems. In practice, teams reach for CoT when a direct one-line answer is more likely to be wrong than a stepwise solution. It can also make model behavior easier to inspect: if the model shows its

papoo.work