Author

aiterms

What is streaming (token streaming)?

Streaming, or token streaming, is when an AI model sends you its output incrementally as it is generated instead of waiting until the whole answer is finished.

Why it matters

This solves the “dead air” problem: users see text quickly, which makes apps feel faster and more interactive. It is especially useful for chatbots, copilots, and any UI where a short time-to-first-token matters more than waiting for the full response.

In practice, teams use streaming when they want:

• faster perceived latency,
• a typing-like effect in the UI,
• earlier partial results for long answers.

How it works

Large language models generate text one token at a time. A token is a chunk of text, often a word piece rather than a whole word. In streaming mode, the server exposes each newly generated token (or small group of tokens) as soon as it is available.

Typically:

1. The client sends a generation request.
2. The model starts decoding tokens.
3. The server pushes partial output back over a streaming channel, such as server-sent events or chunked HTTP.
4. The client appends those chunks to the visible answer until the stream ends.

The important point is that streaming changes delivery, not the model’s core task. The model still generates the same kind of output; you just receive it progressively instead of in one final payload.

Tiny concrete example

Non-streaming response:

{
  "text": "Paris is the capital of France."
}

Streaming response, shown piece by piece:

Paris
 is
 the capital
 of France.

A chat UI might render this as:

Paris is the capital of France.

with the text appearing gradually as the model generates it.

Common pitfalls / when NOT to use it

• Don’t confuse streaming with better quality. It improves responsiveness, not the model’s reasoning.
• Be careful with partial output. Early tokens can be revised by later context in the model’s internal generation process, so your UI should treat streamed text as provisional until the stream ends.
• Not always worth the complexity. For very short outputs, streaming may add implementation overhead without much user benefit.
• Watch for downstream side effects. If you trigger parsing, tool calls, or database writes on partial text, you can create bugs. Usually you wait for completion before acting on the final result.

Related terms

• token
• generation
• latency
• server-sent events
• chat completion

Related terms

• What is batching / continuous batching?
• What is time to first token (TTFT)?
• What is tokens per second (throughput)?
• What is speculative decoding?
• What is an inference endpoint / serving API?

同じ著者の記事

What is speculative decoding?

Speculative decoding is a text-generation trick that lets a fast “draft” model propose several next tokens and then uses a larger “target” model to verify them, often reducing end-to-end latency without changing the final output distribution. Autoregressive LLMs normally generate one token at a time, and each token requires a full forward pass through the big model. That is accurate, but it can be slow. Speculative decoding matters when you want lower latency or higher throughput without switchi

papoo.work

What is tokens per second (throughput)?

Tokens per second, or throughput, is the rate at which a language model generates or processes tokens over time. Throughput is a practical way to describe model speed. If you care about user experience, cost efficiency, or server capacity, tokens per second tells you how much text the system can handle in a given time. You’d use it when comparing models, sizing infrastructure, or setting expectations for streaming responses. In practice, teams often care about both throughput and latency

papoo.work

What is time to first token (TTFT)?

Time to first token (TTFT) is the time between sending a generative AI request and receiving the model’s first output token. TTFT is the part of latency users feel first. If it is high, the system can seem “stuck” even if the full answer eventually streams quickly. You usually care about TTFT when you want: a responsive chat or copilot experience good perceived performance in streaming UIs to compare serving setups, models, or prompts to separate “slow to start” from “slow to finish” In practice

papoo.work

What is batching / continuous batching?

Batching is the practice of grouping multiple requests together so a model can process them more efficiently, and continuous batching is the version where new requests are added to that group while others are still running. Large models are expensive to run because the same GPU is often underused if you serve one request at a time. Batching improves throughput by sharing compute across many requests. Continuous batching goes a step further: it keeps the GPU busy by letting the server dynamically

papoo.work

What is an inference endpoint / serving API?

An inference endpoint, or serving API, is a network service you send inputs to in order to get model predictions back in real time. It solves the problem of turning a trained model into something applications can actually use. You reach for an inference endpoint when you want: a web or mobile app to call a model on demand a backend to classify, rank, summarize, or generate text without running the model itself a standardized way to deploy, scale, monitor, and secure model requests In practice, m

papoo.work

What is quantization?

Quantization is the process of representing numbers with fewer bits, so a model or computation uses less memory and can often run faster. Large neural networks store weights, activations, and sometimes key/value caches as floating-point numbers. That is accurate, but expensive. Quantization reduces that cost by converting some of those values to lower-precision formats, such as 8-bit integers or even 4-bit representations. You usually reach for quantization when you want one or more of these: lo

papoo.work

What is distillation?

Distillation is a way to train a smaller model to imitate a larger, better one, so you keep much of the quality while reducing cost, latency, or memory use. Distillation solves a very practical problem: the model you want may be too slow, too expensive, or too large to deploy everywhere. You reach for distillation when you want one or more of these: lower inference cost faster responses smaller on-device models simpler deployment a model that approximates a stronger system without retraining fro

papoo.work

What is instruction tuning?

Instruction tuning is a way to train a language model to follow natural-language requests more reliably by fine-tuning it on examples of instructions paired with good responses. A base language model is good at predicting text, but that does not automatically make it good at doing what a user asks. Instruction tuning helps bridge that gap. You’d reach for it when you want a model to respond more helpfully to prompts like: “Summarize this email” “Write SQL for this schema” “Explain this code as i

papoo.work

What is RLHF?

RLHF, or reinforcement learning from human feedback, is a way to train a model so its outputs better match what people prefer, not just what is statistically likely. A plain language model learns to predict text. That is useful, but it does not automatically make the model helpful, honest, safe, or aligned with user intent. RLHF is used when you want to steer a model toward human preferences: better answers, fewer toxic outputs, more helpful refusals, and responses that fit product goals. In

papoo.work

What is fine-tuning?

Fine-tuning is the process of taking a pre-trained model and training it a little more on a smaller, task-specific dataset so it behaves better for your use case. Pre-trained models are general-purpose: they know a lot, but not your company’s style, domain vocabulary, or output format. Fine-tuning helps when you want the model to: follow a specific tone or schema consistently, handle niche domain language, improve performance on a narrow task, reduce prompt engineering overhead for repeated work

papoo.work