Author

aiterms

What is time to first token (TTFT)?

Time to first token (TTFT) is the time between sending a generative AI request and receiving the model’s first output token.

Why it matters

TTFT is the part of latency users feel first. If it is high, the system can seem “stuck” even if the full answer eventually streams quickly.

You usually care about TTFT when you want:

• a responsive chat or copilot experience
• good perceived performance in streaming UIs
• to compare serving setups, models, or prompts
• to separate “slow to start” from “slow to finish”

In practice, teams often optimize TTFT before chasing total response time, because a fast first token makes the product feel much more interactive.

How it works

A model does not usually begin emitting text the instant your request arrives. Before the first token appears, several steps may happen:

1. Request handling — the server receives the prompt, validates it, and queues it if needed.
2. Prompt processing — the model encodes the input and prepares internal state.
3. First decode step — the model generates the first output token.
4. Streaming delivery — the token is sent back to the client, which measures the elapsed time.

TTFT is typically measured from the moment the request is sent, or from when the server accepts it, until the first token is received by the client. The exact measurement point can vary by system, so compare TTFT numbers only when the measurement method is the same.

TTFT is different from:

• total latency: time until the whole response is done
• throughput: how many tokens per second the system can produce after it starts
• inter-token latency: delay between later tokens in a stream

Tiny concrete example

A user asks a coding assistant:

User: "Rewrite this function to be async."

The assistant starts streaming:

• 180 ms: first token arrives: "Sure,"
• 220 ms: second token arrives: "here"
• 240 ms: third token arrives: "is"

Here, the TTFT is 180 ms. The rest of the answer may still take much longer, but the user already sees that the system is alive.

Common pitfalls / when NOT to use it

• Don’t treat TTFT as the whole performance story. A low TTFT with very slow token generation can still feel bad.
• Don’t compare numbers from different measurement setups. Client-side timing, server-side timing, queueing, and network conditions can all change the result.
• Don’t optimize TTFT alone if the response is tiny. For short outputs, total latency may matter more.
• Don’t assume it is always the right KPI. For batch jobs or offline generation, throughput and total completion time are often more important.

If you are building a streaming chat UX, TTFT is usually one of the best metrics to watch. If you are generating long documents or running offline inference, it may be less important than overall throughput.

Related terms

• What is streaming (token streaming)?
• What is batching / continuous batching?
• What is tokens per second (throughput)?
• What is speculative decoding?
• What is an inference endpoint / serving API?

同じ著者の記事

What is speculative decoding?

Speculative decoding is a text-generation trick that lets a fast “draft” model propose several next tokens and then uses a larger “target” model to verify them, often reducing end-to-end latency without changing the final output distribution. Autoregressive LLMs normally generate one token at a time, and each token requires a full forward pass through the big model. That is accurate, but it can be slow. Speculative decoding matters when you want lower latency or higher throughput without switchi

papoo.work

What is tokens per second (throughput)?

Tokens per second, or throughput, is the rate at which a language model generates or processes tokens over time. Throughput is a practical way to describe model speed. If you care about user experience, cost efficiency, or server capacity, tokens per second tells you how much text the system can handle in a given time. You’d use it when comparing models, sizing infrastructure, or setting expectations for streaming responses. In practice, teams often care about both throughput and latency

papoo.work

What is streaming (token streaming)?

Streaming, or token streaming, is when an AI model sends you its output incrementally as it is generated instead of waiting until the whole answer is finished. This solves the “dead air” problem: users see text quickly, which makes apps feel faster and more interactive. It is especially useful for chatbots, copilots, and any UI where a short time-to-first-token matters more than waiting for the full response. In practice, teams use streaming when they want: faster perceived latency, a typing-lik

papoo.work

What is batching / continuous batching?

Batching is the practice of grouping multiple requests together so a model can process them more efficiently, and continuous batching is the version where new requests are added to that group while others are still running. Large models are expensive to run because the same GPU is often underused if you serve one request at a time. Batching improves throughput by sharing compute across many requests. Continuous batching goes a step further: it keeps the GPU busy by letting the server dynamically

papoo.work

What is an inference endpoint / serving API?

An inference endpoint, or serving API, is a network service you send inputs to in order to get model predictions back in real time. It solves the problem of turning a trained model into something applications can actually use. You reach for an inference endpoint when you want: a web or mobile app to call a model on demand a backend to classify, rank, summarize, or generate text without running the model itself a standardized way to deploy, scale, monitor, and secure model requests In practice, m

papoo.work

What is quantization?

Quantization is the process of representing numbers with fewer bits, so a model or computation uses less memory and can often run faster. Large neural networks store weights, activations, and sometimes key/value caches as floating-point numbers. That is accurate, but expensive. Quantization reduces that cost by converting some of those values to lower-precision formats, such as 8-bit integers or even 4-bit representations. You usually reach for quantization when you want one or more of these: lo

papoo.work

What is distillation?

Distillation is a way to train a smaller model to imitate a larger, better one, so you keep much of the quality while reducing cost, latency, or memory use. Distillation solves a very practical problem: the model you want may be too slow, too expensive, or too large to deploy everywhere. You reach for distillation when you want one or more of these: lower inference cost faster responses smaller on-device models simpler deployment a model that approximates a stronger system without retraining fro

papoo.work

What is instruction tuning?

Instruction tuning is a way to train a language model to follow natural-language requests more reliably by fine-tuning it on examples of instructions paired with good responses. A base language model is good at predicting text, but that does not automatically make it good at doing what a user asks. Instruction tuning helps bridge that gap. You’d reach for it when you want a model to respond more helpfully to prompts like: “Summarize this email” “Write SQL for this schema” “Explain this code as i

papoo.work

What is RLHF?

RLHF, or reinforcement learning from human feedback, is a way to train a model so its outputs better match what people prefer, not just what is statistically likely. A plain language model learns to predict text. That is useful, but it does not automatically make the model helpful, honest, safe, or aligned with user intent. RLHF is used when you want to steer a model toward human preferences: better answers, fewer toxic outputs, more helpful refusals, and responses that fit product goals. In

papoo.work

What is fine-tuning?

Fine-tuning is the process of taking a pre-trained model and training it a little more on a smaller, task-specific dataset so it behaves better for your use case. Pre-trained models are general-purpose: they know a lot, but not your company’s style, domain vocabulary, or output format. Fine-tuning helps when you want the model to: follow a specific tone or schema consistently, handle niche domain language, improve performance on a narrow task, reduce prompt engineering overhead for repeated work

papoo.work