#ai-glossary

Meta-prompting is the practice of using one prompt to design, improve, or control another prompt, rather than asking the model to do the task directly. It helps when a plain prompt is too vague, too brittle, or too hard to maintain. Instead of hand-tuning a long instruction by trial and error, you can ask an LLM to generate, refine, critique, or transform prompts for you. In practice, teams reach for meta-prompting when they want: better prompt quality with less manual iteration, reusable prompt

Tree-of-thought prompting, often abbreviated ToT, is a prompting method where an LLM explores multiple reasoning branches instead of committing to a single straight-line answer. Tree-of-thought prompting is useful when a problem benefits from deliberate search: puzzle solving, planning, multi-step reasoning, or tasks where an early mistake can derail the whole answer. In practice, it gives the model a way to "think in branches," then compare candidates before choosing one. You'd reach for To

Self-consistency is a way to get a language model to answer by generating several reasoning paths and then choosing the answer that appears most often, instead of trusting a single chain of thought. A single reasoning trace from an LLM can be brittle: one unlucky step can derail the final answer. Self-consistency helps when the task needs multi-step reasoning, such as math word problems, logic puzzles, or multi-hop question answering. In practice, you reach for it when you want a cheap reliabili

Using XML tags or other delimiters in prompts means wrapping parts of your prompt in clear markers like `<context>...</context>` so the model can separate instructions, data, and examples more reliably. This pattern helps when a prompt contains multiple kinds of content at once: instructions, source text, user input, examples, or fields you want the model to treat differently. Delimiters reduce ambiguity and make prompts easier to read, debug, and reuse. In practice, teams reach for XML-style ta

A prompt template is a reusable prompt with placeholders for variable details, so you can generate consistent LLM inputs without rewriting the whole prompt each time. Prompt templates solve a simple but common problem: most prompts are partly fixed and partly changing. In practice, they help you: keep instructions consistent across many requests swap in user-specific data, documents, or task details reduce copy-paste errors when building apps, workflows, or evals make prompts easier to version,

Structured output, often called JSON mode, is a way to ask an LLM to return data in a fixed machine-readable format instead of free-form text. When you want an answer that another program can safely parse, plain chat text is fragile. A model might add extra words, change field names, or wrap the result in an explanation. Structured output helps when you need: predictable API responses reliable extraction into downstream systems cleaner automation for forms, workflows, or tools outputs that match

Prompt engineering is the practice of designing and refining the text you give to a language model so it produces more useful, reliable, and specific outputs. Large language models are sensitive to how you ask. A vague request can produce a vague answer; a well-structured prompt can improve format, scope, tone, and task performance. You reach for prompt engineering when you want to: get a model to follow instructions more consistently, steer it toward a role, style, or output format, reduce back

A system prompt is the instruction text that sets the assistant’s role, behavior, and boundaries before the user’s message is processed. System prompts help you control how an LLM behaves without changing the model itself. They’re useful when you want consistent tone, safety rules, formatting, or task-specific behavior across many requests. In practice, teams use system prompts to: keep outputs on-brand, enforce policies or guardrails, define what the assistant should and should not do, reduce a

Few-shot prompting, also called in-context learning, is a way of asking a language model to do a task by giving it a few example inputs and outputs in the prompt. Few-shot prompting helps when you want the model to follow a pattern, format, or classification rule without training or fine-tuning a new model. You’d reach for it when: the task is simple but the output needs to be consistent, you can show the model 1–5 good examples, you want to prototype quickly before investing in fine-tuning

Zero-shot prompting is asking an AI model to do a task without giving it any worked examples first. It is the simplest way to use a large language model: you describe the task in plain language and let the model infer the pattern from its pretraining. That makes it useful when you want: a fast baseline before writing better prompts simple classification, extraction, or drafting tasks lower prompt length and less example curation a test of whether the model already “gets” the task well enough In

Chain-of-thought prompting, often abbreviated CoT, is a way of asking a language model to produce intermediate reasoning steps before giving the final answer. It is used to improve performance on tasks that benefit from multi-step reasoning, such as arithmetic, logic, and some question-answering problems. In practice, teams reach for CoT when a direct one-line answer is more likely to be wrong than a stepwise solution. It can also make model behavior easier to inspect: if the model shows its

Context engineering is the practice of deliberately assembling the right information, instructions, tools, and examples into an AI model’s context so it can do a task reliably. Large language models do not “know” what you want unless you provide it in the prompt, surrounding messages, retrieved documents, tool outputs, or examples. Context engineering helps you get better accuracy, fewer hallucinations, and more consistent behavior by shaping what the model sees before it answers. In practice, i

A context window is the amount of text a language model can read and keep in mind at once while it generates a response. The context window sets the practical limit for what you can ask a model to consider in a single turn: your prompt, system instructions, chat history, retrieved documents, and the model’s own recent output all compete for that space. In practice, it matters when you want the model to: follow long instructions, answer questions about a large document, maintain conversation stat

Prompt caching is a way to reuse the cost and latency of processing the same prompt prefix instead of sending or recomputing it from scratch every time. If your app sends a long, repeated system prompt, policy block, tool schema, or document context on many requests, you can end up paying to re-process the same text over and over. Prompt caching helps reduce that waste. In practice, teams reach for it when: a large part of the prompt stays the same across requests, they want lower latency for re

Prompt injection is when untrusted text causes an AI model or agent to follow attacker-controlled instructions instead of the user’s intended task. Prompt injection matters because modern LLM apps often mix trusted instructions with untrusted content: web pages, emails, documents, tickets, chat messages, or tool outputs. If the model treats that content as instructions, it can be tricked into leaking data, ignoring policies, or taking unintended actions. You usually care about prompt injection w

RAG evaluation is the process of checking whether a retrieval-augmented generation system’s answer is faithful to the provided context and relevant to the user’s question. RAG systems are meant to answer with help from retrieved documents, not just from the model’s memory. Evaluation tells you whether the system is actually doing that well. In practice, teams use RAG evaluation to catch two common failure modes: Low faithfulness: the answer sounds plausible but is not supported by th

RAG (retrieval-augmented generation) and fine-tuning are two different ways to make a language model more useful: RAG gives the model external information at answer time, while fine-tuning changes the model itself using training data. If your app needs accurate answers about fast-changing or private information, RAG is usually the first thing to try because it can use your documents without retraining the model. If you want the model to consistently follow a format, style, or task pattern, fine-

Chunking is the practice of splitting a larger piece of text or data into smaller pieces so a system can process, store, or retrieve it more effectively; semantic chunking does the same thing, but tries to cut at meaningful boundaries instead of by fixed length. Chunking is a core technique in retrieval-augmented generation (RAG), document search, summarization, and any workflow that needs to feed long content into an LLM. Why teams use it: LLMs have context limits, so very large documents must

Semantic search is a way to find information by meaning, not just by matching the exact words you typed. Traditional keyword search is good when you know the exact term to look for. Semantic search helps when users phrase the same idea in different words, use synonyms, or ask a question instead of naming a document. You’d reach for it when: search queries are messy or conversational, exact wording varies a lot across documents, you want to retrieve “about the same thing” rather than “contains th

An embedding is a vector embedding: a list of numbers that represents something—like a word, sentence, image, or user—in a way a model can compare mathematically. Embeddings turn messy, human data into a form machines can use for: Similarity search: “Find documents like this one.” Recommendations: “Show items close to what the user liked.” Clustering and classification: “Group related things together.” Retrieval for LLMs: “Fetch the most relevant context before generating an

A vector database, also called a vector store, is a system for storing and searching data by its numerical embeddings so you can find items that are “most similar” rather than exactly matching text or IDs. Regular databases are great when you know the exact thing you want: a user ID, a product SKU, a timestamp range. But many AI features need semantic search or nearest-neighbor retrieval: “find questions like this one,” “retrieve passages relevant to this prompt,” or “recommend items

Retrieval-augmented generation, or RAG, is a way to make a language model answer questions using relevant external documents it retrieves first, instead of relying only on what it memorized during training. RAG helps when you want answers that are: grounded in your own content, such as docs, policies, tickets, or product knowledge easier to update than retraining a model more likely to mention current or proprietary information In practice, teams reach for RAG when a model needs to answer “what

Hybrid search is a retrieval method that combines keyword search with vector similarity search so you can find documents that match both the exact words a user typed and the broader meaning of the query. Pure keyword search is good when the exact term matters. Pure vector search is good when wording varies but meaning is similar. In practice, many real queries need both. You reach for hybrid search when: users may search with exact product names, error codes, or technical terms the same

A reranker, often implemented as a cross-encoder, is a model that takes a query and a candidate result together and scores how well they match. A reranker solves the “good enough first pass, better final answer” problem. In search and retrieval systems, you usually start with a fast retriever that finds a few dozen or few hundred candidate documents. That first stage is optimized for speed, not perfect ranking. A reranker then looks at those candidates more carefully and reorders them so the mos

Top-k retrieval is a way to search a collection and return the k most relevant items for a query, instead of trying to sort or inspect everything. Top-k retrieval is the standard first step in search, recommendation, and retrieval-augmented generation (RAG). You use it when you want a fast shortlist: the most relevant documents, passages, products, images, or candidates, without paying the cost of full ranking over the entire corpus. In practice, teams reach for top-k retrieval when: the dat

A knowledge base for RAG is the collection of documents or records a system retrieves from to ground an LLM’s answer in relevant source material. RAG, short for retrieval-augmented generation, works best when the model can search a useful, trustworthy set of content before it writes. That content is the knowledge base. You’d reach for one when the model needs to answer questions from: company docs support articles policies and handbooks product specs research notes tickets or tickets-like record

Contextual chunking, also called late chunking, is a way to split a document into retrieval chunks after a model has already encoded the whole document, so each chunk keeps some awareness of the surrounding context. Normal chunking cuts text first and then embeds each piece alone. That is simple, but it can make chunks semantically thin: a paragraph may be ambiguous without the surrounding sections, headings, or earlier definitions. Late chunking helps when you want better retrieval quality

A vector index is a data structure that helps a system quickly find the most similar embeddings or vectors to a query vector; it is also commonly called an ANN index, short for approximate nearest neighbor index. If you store thousands, millions, or billions of vectors and want “find the closest ones,” a brute-force scan is usually too slow. A vector index lets you trade a little exactness for much faster retrieval, which is why it shows up in search, retrieval-augmented generation (RAG), recomm

Cosine similarity is a way to measure how similar two vectors are by comparing the angle between them, not their size. It’s useful when you care more about direction than magnitude. In AI and information retrieval, that usually means comparing embeddings, text vectors, or feature vectors to find items that “mean the same thing” even if they have different lengths. You’d reach for cosine similarity when: ranking search results by semantic closeness, comparing document, sentence, or image embe

Metadata filtering in retrieval is the practice of narrowing a search or vector search to documents that match structured fields like date, author, source, tenant, language, or content type before or during ranking. It solves a very practical problem: not every relevant item is relevant for this user, this time, or this task. If you are building search or retrieval-augmented generation (RAG), metadata filters help you: exclude clearly irrelevant documents early, enforce access control or tenant

Query rewriting or query expansion is the process of changing a user’s search query into a better one—usually by adding, removing, or rephrasing terms so a search or retrieval system can find more relevant results. People often search with short, vague, or underspecified queries. Systems that use the raw query alone may miss relevant documents because the wording does not match the target content. Query rewriting/expansion helps when you want better recall, better intent matching, or more robust

The lost-in-the-middle problem is the tendency of a language model to miss information that appears in the middle of a long context, even when that information is relevant to the answer. This matters any time you stuff a lot of text into an LLM prompt: long documents, chat histories, codebases, retrieval-augmented generation (RAG), or multi-step agent traces. In practice, it means: the model may answer well using the beginning or end of the context, but fail on facts buried in the middle, so

Graph RAG, short for Graph Retrieval-Augmented Generation, is a way to help an LLM answer questions by retrieving information from a graph of connected entities and relationships instead of, or in addition to, plain text chunks. Classic RAG works well when the answer lives in a few relevant text passages. But some questions are really about relationships: who depends on whom, how events connect, which product is linked to which system, or how a concept appears across many documents.

Jailbreaking is the act of tricking a model or device into bypassing its normal safety, policy, or security restrictions. For AI systems, jailbreaking is usually about getting a model to ignore refusal rules, reveal restricted information, or produce disallowed content. For devices and operating systems, “jailbreaking” means removing vendor-imposed limits so you can install unapproved software or gain deeper system access. In practice, the AI meaning is the one most developers mean today. It mat

A document loader or parser is a component that takes a file or data source and turns it into text and metadata an AI system can actually use. LLMs do not read PDFs, Word files, HTML pages, or emails directly in their native form. If you want to do retrieval-augmented generation (RAG), search, summarization, or extraction over real-world documents, you first need a reliable way to ingest them. That is where loaders and parsers come in: A loader fetches content from somewhere: local files, cl

Tool error handling and retries are the ways an AI agent responds when a tool call fails, by classifying the failure, deciding whether it is safe to try again, and then either retrying with a better request or stopping and surfacing the error. Tools fail for ordinary reasons: network timeouts, rate limits, malformed inputs, missing permissions, empty results, or temporary backend outages. If an agent treats every failure the same, it either gives up too early or blindly repeats a bad request. Go

A code execution tool, or sandbox, is a controlled runtime that lets an AI agent run code safely without giving it full access to your systems. LLMs are good at writing code, but they cannot reliably verify code by thought alone. A sandbox lets an agent: run generated code check outputs inspect errors iterate on fixes You’d reach for it when the task needs computation, file manipulation, data transformation, or testing. In practice, teams use sandboxes when “just answer in text” is too britt

A computer-use or browser-use tool is a capability that lets an AI agent operate software directly—usually a web browser, and sometimes the broader desktop—by clicking, typing, scrolling, and reading the visible screen instead of calling an API. Not every task has a clean API. Many real workflows still live in web apps: logging into dashboards, filling forms, checking inventories, downloading reports, or copying data between systems. A browser-use tool helps an LLM or agent handle those tasks by

A tool result or tool message is the message an app sends back to the model after the model asks it to use an external tool, such as a search API, calculator, database query, or function call. LLMs do not reliably know fresh data, private data, or exact computations on their own. A tool result lets the system return the tool’s output to the model in a structured way so the model can continue the conversation with real evidence. You’d use it whenever the model needs to: fetch data from an

Parallel tool calling is when an LLM asks for multiple tools to be run in the same turn, so independent actions can happen at the same time instead of one after another. Use parallel tool calling when the model needs several unrelated results, such as fetching weather for three cities, looking up multiple records, or calling a search API and a calculator in the same step. It matters because it can reduce latency and keep agent flows simpler: rather than forcing the model to serialize independent

A JSON schema for tool arguments is a machine-readable description of what inputs a tool expects, including the names, types, and constraints of its parameters. When an LLM calls a tool, it needs a reliable way to format the arguments so the tool can actually run. A JSON schema helps by telling the model and the application: which fields are required what type each field should be whether values must match a pattern, range, or enum how to validate the request before executing the tool In practic

A tool schema or tool definition is a machine-readable description of a function, API, or action an LLM can call, including its name, what it does, and the inputs it accepts. Tool schemas let you connect a model to real capabilities without letting it invent the interface. In practice, they solve three common problems: Structured calling: the model can ask for a function call instead of writing brittle free-form text. Validation: the runtime can check that the model supplied the

Function calling, also called tool calling or structured tool use, is a way for an AI model to ask your app to run a specific function with specific arguments instead of guessing the answer itself. Large language models are good at language, but they are not reliable sources of fresh facts, private data, or side effects like “send the email” or “look up today’s weather.” Function calling solves that by letting the model produce a structured request that your code can execute. You reach f

The Model Context Protocol (MCP) is an open standard that lets AI apps connect to external tools, data sources, and workflows through a common interface, usually via an MCP client talking to an MCP server. Without a shared protocol, every AI app tends to build one-off integrations for files, databases, APIs, and internal tools. MCP solves that by giving model-powered applications a standard way to discover and use capabilities from different systems. In practice, you reach for MCP when y

An MCP server is a service that exposes tools, data, or actions to an AI client using the Model Context Protocol, so an LLM-powered app can safely discover and call external capabilities in a standard way. Without a standard protocol, every AI app ends up integrating with files, databases, APIs, and internal tools in its own custom way. That makes integrations harder to build, harder to reuse, and harder to govern. An MCP server solves that by giving AI clients a common interface for: discoverin

An MCP client is the software that connects an app or agent to an MCP server so it can discover and use external tools, resources, or prompts through the Model Context Protocol. If you want an LLM app to safely call APIs, read files, query data, or use other external capabilities in a standardized way, you need something on the client side to speak MCP. In practice, the MCP client is the piece that: opens and manages the connection to one or more MCP servers, asks what capabilities those servers

An MCP transport is the way a Model Context Protocol client and server exchange messages, most commonly either over standard input/output (`stdio`) or over HTTP-based connections. MCP defines *what* messages are sent; the transport defines *how* they move between processes or over a network. You choose a transport based on deployment shape: `stdio` is the usual choice for a local, single-user tool or desktop app starting a server as a child process. HTTP is the usual choice when the MCP

An MCP resource is a piece of data that a server exposes to a model or client through the Model Context Protocol, usually as a named, readable thing like a file, document, or other context object. Resources are how MCP lets an assistant work with context that already exists somewhere else without copying it into the prompt by hand. In practice, they are useful when you want an LLM app to inspect docs, logs, configs, or other stored information in a structured way. You’d reach for resourc

An MCP prompt is a prompt template or prompt resource exposed through the Model Context Protocol so an app or agent can fetch and use a standardized prompt from a server instead of hard-coding it. MCP is designed to let tools, models, and clients work together through a common protocol. Prompts are one part of that: they let a server publish reusable instructions, examples, or task templates that a client can retrieve dynamically. You’d use an MCP prompt when you want: a consistent prompt across

Tool choice (often exposed as `tool_choice`) is the setting that tells an LLM whether it may decide on its own to call a tool, must call a tool, or must not call any tool. Tool choice lets you control how an assistant interacts with external capabilities like search, databases, calculators, or internal APIs. You reach for it when you want to: let the model answer directly unless a tool is clearly needed, force a tool call for workflows that must use fresh or trusted data, or prevent tool use

A tool call ID is a unique identifier attached to one specific tool invocation so the model and your application can refer to that exact call again later. Tool calls are often part of a multi-step exchange: the model asks for a function or external service to run, your app executes it, and then you send the result back. The tool call ID is what keeps that round-trip unambiguous. You reach for it when: a model makes more than one tool call in the same turn you need to match each tool result to th

A tool registry is a catalog that lists the tools an AI agent or application can use, along with the metadata needed to call them correctly. When you build an agent that can search, fetch data, send messages, or run internal workflows, the model needs more than just a tool name. It needs to know: what each tool does, what inputs it expects, how to call it safely, and often which tools are available in the current context. A tool registry solves that discovery problem. Instead of hard-coding

A stateless agent makes each decision from the current input only, while a stateful agent carries useful information across steps so later decisions can depend on earlier ones. This distinction affects how you design AI agents, chatbots, workflow automations, and any system that acts over multiple turns or tasks. Use a stateless design when each request is independent, easy to retry, and simple to scale. Use a stateful design when the agent must remember goals, intermediate resul

Agent context compaction, also called context summarization, is the practice of shrinking an AI agent’s conversation or working memory into a shorter representation so it can keep going without exceeding the model’s context window. Agents often run for many turns, call tools, inspect files, and accumulate lots of intermediate state. That history eventually becomes too large to feed back into the model. Context compaction solves two practical problems: Keeps long-running agents usable whe

A supervisor agent, also called a routing agent, is an agent that decides which specialist tool, workflow, or sub-agent should handle a user’s request instead of trying to do everything itself. A single general-purpose agent can get messy when a task spans multiple skills: search, code, database access, summarization, planning, and verification. A supervisor/router agent helps by directing each request to the right worker, which can make systems easier to maintain, safer to control, and cheaper

Agent autonomy is the degree to which an AI agent can decide and act on its own, from simply suggesting actions to fully executing a task with little or no human intervention. “Autonomy” is the main design choice when building agentic systems. It tells you how much responsibility the system has versus the human operator. You reach for higher autonomy when: the task is repetitive, bounded, and low risk speed matters more than fine-grained human control the environment changes often enough that ha

An agent scratchpad is the temporary working area an AI agent uses to keep intermediate notes, steps, or partial results while it solves a task. A scratchpad helps an agent “think through” a multi-step problem instead of trying to answer in one shot. It is useful when a task requires planning, tool use, multi-step reasoning, or keeping track of state across turns. In practice, teams reach for a scratchpad when they want the agent to: break a task into steps, remember what it has already trie

A subagent is a smaller agent that a larger agent delegates a specific task to, instead of doing everything itself. Subagents help break a complex job into manageable pieces. In practice, teams reach for them when one agent would otherwise have to plan, execute, check, and summarize all at once. They are useful when you want: specialization: one agent handles research, another writes code, another verifies results better control: the parent agent can supervise or combine outputs simpler prompts:

An orchestrator-worker agent pattern is a setup where one lead agent plans, delegates, and coordinates work, while several worker agents or tools handle smaller tasks in parallel or in sequence. This pattern helps when a single LLM call is not enough because the job is too broad, too long, or naturally decomposes into parts. The lead agent reduces the problem into sub-tasks, assigns them, and then combines the results. In practice, you reach for it when you want: better structure for complex req

Reflection, or self-critique, is when an AI agent reviews its own output or plan, spots mistakes or weaknesses, and then revises itself before acting. Agentic systems can make more than one kind of error: they may miss a constraint, choose a bad tool, hallucinate a fact, or produce a plan that looks plausible but won’t work. Reflection adds a built-in “check yourself” step, which is useful when you want better reliability than a single pass from the model. In practice, teams reach for reflection

Task decomposition is the practice of breaking a larger goal into smaller, manageable sub-tasks that can be solved one by one. Complex tasks are easier to plan, execute, and debug when they are split into parts. In AI systems, software projects, and agent workflows, decomposition helps you: reduce ambiguity in a broad request assign work to different components or steps catch errors earlier by checking intermediate results make progress on problems that are too large to solve in one shot In prac

A multi-agent system is a setup where multiple autonomous agents work separately or together to complete a task, often with some form of coordination or agent orchestration. A single agent is often enough for simple tasks, but multi-agent systems help when the problem is naturally split into parts: research, planning, execution, review, or parallel exploration. They can also let you assign different roles or tools to different agents, which is useful when one model instance would be too slow, to

Agent memory is the information an AI agent keeps so it can keep working across turns, and it usually means two different things: short-term memory for the current task context, and long-term memory for stored facts or past interactions it can retrieve later. Without memory, an agent is basically reset every time you ask it something new. That makes it bad at multi-step tasks, personalization, follow-up questions, and anything that needs continuity. In practice, memory helps an agent: re

A planner-executor agent is an AI system that first makes a plan, then carries out that plan step by step, usually by delegating the “thinking ahead” to one component and the “doing” to another. This pattern is useful when a task is too complex to solve in one shot. Instead of asking a model to answer, browse, call tools, and self-correct all at once, you split the work into two roles: Planner: decides the sequence of steps, subgoals, or tool calls. Executor: follows those steps and prod

Human-in-the-loop for agents means a person stays in the decision loop to review, approve, correct, or stop an AI agent before it takes important actions. Agentic systems can plan, call tools, and act across multiple steps. That is useful, but it also creates risk: a wrong tool call, a bad assumption, an unsafe side effect, or an action the user never intended. Human-in-the-loop adds a control point so the agent does not fully act on its own when the cost of a mistake is high. You reach for it w

An agent harness or agent runtime is the software layer that runs an AI agent, manages its loop of thinking and acting, and connects it to tools, memory, and external systems. An LLM by itself just returns text. An agent needs more than that: it has to decide when to call a tool, pass results back into the model, stop at the right time, handle errors, and keep track of state. That is what the harness/runtime does. It is the scaffolding around the model that turns “generate a response” in

An AI agent, also called an LLM agent or autonomous agent, is a system that can pursue a goal by deciding what to do next, taking actions, and using the results to keep going. Most AI systems just answer a prompt once. An agent is useful when one answer is not enough and the system needs to act over multiple steps: search, call tools, inspect results, retry, and stop when the goal is met. You’d reach for an agent when the task is: open-ended, not a single fixed response dependent on external

An agentic workflow is a workflow where an AI system can decide and take multi-step actions toward a goal, instead of only producing a one-shot answer; you may also hear this called an agentic system. Use an agentic workflow when the task is not just “generate text,” but “figure out what to do next.” That includes tasks like investigating an issue, pulling together information from multiple tools, retrying after failure, or following conditional branches based on intermediate results

A ReAct agent is an LLM-driven system that alternates between reasoning and acting—often called reason + act or ReAct prompting—so it can decide what to do next, use tools, and update its plan from the results. ReAct is useful when a model should not just produce an answer, but also take steps in the world: search a document set, call an API, inspect a database, or ask for more information. In practice, teams reach for it when: a plain chat completion is too static, the task

The agent loop, also called the perceive-plan-act loop, is the repeated cycle where an AI agent observes its situation, decides what to do next, and then takes an action. This is the basic control pattern behind many agentic systems. Instead of asking a model for one one-off answer, you let it keep checking the world, updating its plan, and acting until the task is done or it should stop. You reach for it when the task is dynamic, multi-step, or depends on intermediate results: calling tools, br

Tool use in AI agents is when an LLM-based agent can call external functions, APIs, or programs—often called a tool calling agent—to get information or take actions it cannot do reliably from text alone. A plain language model is good at generating text, but it does not reliably know live facts, cannot directly query a database, and should not guess at actions like sending email or creating tickets. Tool use solves that by letting the model delegate specific steps to purpose-built systems. Y

Grounding is the practice of connecting an AI model’s output to a real source of truth, so its answer is based on evidence instead of only pattern-matched text. Large language models can sound confident even when they are wrong. Grounding helps reduce that by tying responses to something verifiable: a document, database, tool result, sensor reading, or other external source. You reach for grounding when correctness matters more than fluency—customer support, enterprise search, policy answers, pr

Top-p sampling, also called nucleus sampling, is a way to make a language model pick the next token from only the smallest set of likely options whose combined probability reaches a chosen threshold. If you always choose the single most likely next token, model outputs can become repetitive and brittle. If you sample from all tokens, outputs can get noisy and incoherent. Top-p gives you a middle ground: it keeps the model flexible, but trims away the long tail of very unlikely tokens before samp

Temperature is a sampling setting that controls how random or deterministic an LLM’s next-word choices are when it generates text. Temperature is one of the simplest ways to trade off consistency versus variety. Use a lower temperature when you want stable, repeatable answers: classification-like outputs, structured data, summaries, or code. Use a higher temperature when you want more diversity: brainstorming, creative writing, alternate phrasings, or exploratory chat. In practice, many

Attention, or self-attention, is a way for a model to decide which other words or tokens in the same input matter most when interpreting each token. Self-attention solves a basic language problem: the meaning of a word often depends on other words nearby or far away. It lets a model build a context-aware representation of each token instead of treating tokens mostly independently or relying only on fixed-size windows. In practice, this is why Transformers work so well for text, code, and other s

A transformer is a neural network architecture that processes sequences by using attention to compare each token with the others, making it a flexible foundation for modern language models and many other sequence tasks. Before transformers, many sequence models had to read text step by step, which made long-range dependencies harder to handle and training slower to parallelize. Transformers solve this by letting the model look at many positions in the input at once. In practice, you reach for a

A token is a chunk of text that a language model reads and predicts, and tokenization is the process of splitting text into those chunks. LLMs do not operate directly on raw characters or words; they operate on tokens. That matters because: Input limits are measured in tokens, not characters. Cost and latency are often tied to token count in LLM APIs. Model quality depends on how text is tokenized: the same sentence can become very different token sequences across tokenizers.

A mixture-of-experts (MoE) model is a neural network that routes each input, or part of an input, to only a small subset of specialized submodels (“experts”) instead of using the whole model every time. MoE models are useful when you want much larger capacity without paying the full compute cost on every token or example. In practice, that means you can scale model size to improve quality while keeping inference and training more efficient than a dense model of similar total parameter count.

A parameter is a learned number inside a model, and “model size” usually means how many of those learned numbers the model has. Parameters are the part of a neural network that gets updated during training, so they determine how much capacity the model has to fit patterns in data. You’ll care about parameter count when you want to compare models, estimate training cost, understand memory use, or get a rough sense of how much hardware a model needs. In practice, “bigger” often means more capable,

The KV cache, or key-value cache, is a memory buffer that stores past attention keys and values so a transformer can generate the next token without recomputing everything from scratch. In autoregressive generation, each new token depends on all previous tokens. Without a KV cache, the model would repeatedly re-run attention over the entire prefix at every step, which is much slower than necessary. You reach for a KV cache when you want faster text generation, lower latency, or higher throughput

A hallucination is when an AI system generates output that sounds plausible but is wrong, unsupported, or made up. Hallucinations matter because they break trust. If you use a model for search, summarization, coding help, or customer support, a confident but false answer can mislead users, create bugs, or spread bad information. In practice, teams care about hallucinations whenever they need factual accuracy, source grounding, or reliable behavior. For creative writing, they may be less of a pro

Observability for LLM apps is the practice of collecting structured logs, traces, metrics, and feedback so you can see what an AI application actually did, why it behaved that way, and where it failed. LLM apps are harder to debug than traditional software because the “logic” is partly produced at runtime by a model. A user sees a bad answer, but the cause could be the prompt, retrieved context, tool output, model behavior, latency, token limits, or a broken chain of steps. Observability hel

LLM-as-a-judge is the practice of using a large language model to evaluate or rank outputs from another model, system, or workflow, instead of relying only on human reviewers or fixed metrics. It solves a common problem: many AI outputs are hard to score with simple automated checks. If you are judging a summary for usefulness, a chatbot answer for helpfulness, or two candidate responses for preference, exact-match metrics often miss the point. Teams reach for LLM-as-a-judge when they need: **Fa

An LLM eval, or evaluation, is a test or set of tests used to measure how well a large language model behaves on a task or under a specific set of conditions. If you build with LLMs, you need a way to answer practical questions like: Does this prompt change improve answer quality? Is the model following instructions more reliably? Did a new model version get worse on safety, latency, or factuality? An eval gives you a repeatable way to compare models, prompts, retrieval setups, or fine-tunes. In

A golden dataset or eval set is a small, carefully chosen set of examples with trusted labels or expected outputs that you use to judge whether a model, prompt, or system is working correctly. If you are building an LLM app, an agent, or a classic ML model, you need some repeatable way to answer: “Did this change make things better or worse?” A golden dataset gives you that baseline. Teams use it to: compare model versions or prompts catch regressions before deployment measure qualit

A guardrail is a rule, check, or control that keeps an AI system from producing or doing something unsafe, incorrect, or out of policy. Large language models and agents can generate useful outputs, but they can also hallucinate, leak sensitive data, follow malicious instructions, or take actions you did not intend. Guardrails help reduce that risk. You reach for guardrails when you need the system to stay within boundaries: for example, only answer from approved sources, refuse certain requests,

Hallucination detection is the process of spotting when an AI model’s answer is likely unsupported, false, or made up rather than grounded in the available evidence. Large language models can produce fluent answers that sound right even when they are wrong. Hallucination detection helps reduce the risk of shipping misleading outputs in search, customer support, medical/legal workflows, internal assistants, and any system where users may trust the model too much. In practice, teams use it when th

Regression testing for prompts is the practice of rerunning a fixed set of prompt-based checks after you change a prompt, model, tool, or surrounding code to make sure the system still behaves the way you expect. Prompted LLM systems are brittle in a different way than traditional software: a small wording change, a model upgrade, or a new tool call can improve one case and break another. Regression testing helps you catch those accidental behavior changes before users do. You’d reach for it whe

A base model is a pretrained language model that predicts and generates text, while an instruct model is the same kind of model further tuned to follow human instructions more reliably. This distinction helps you choose the right model for the job: Base models are useful when you want raw next-token generation, custom fine-tuning, or maximum flexibility for downstream tasks. Instruct models are useful when you want the model to answer questions, follow prompts, summarize, write c

LoRA, or low-rank adaptation, is a way to adapt a large pretrained model by training a small number of extra parameters instead of updating all of the model’s weights. Fine-tuning a large model the traditional way can be expensive in memory, storage, and training time. LoRA solves that by letting you specialize a model for a task while keeping the original model frozen. You’d reach for LoRA when you want: lower training cost than full fine-tuning multiple task-specific variants of the same base

Fine-tuning is the process of taking a pre-trained model and training it a little more on a smaller, task-specific dataset so it behaves better for your use case. Pre-trained models are general-purpose: they know a lot, but not your company’s style, domain vocabulary, or output format. Fine-tuning helps when you want the model to: follow a specific tone or schema consistently, handle niche domain language, improve performance on a narrow task, reduce prompt engineering overhead for repeated work

RLHF, or reinforcement learning from human feedback, is a way to train a model so its outputs better match what people prefer, not just what is statistically likely. A plain language model learns to predict text. That is useful, but it does not automatically make the model helpful, honest, safe, or aligned with user intent. RLHF is used when you want to steer a model toward human preferences: better answers, fewer toxic outputs, more helpful refusals, and responses that fit product goals. In

Instruction tuning is a way to train a language model to follow natural-language requests more reliably by fine-tuning it on examples of instructions paired with good responses. A base language model is good at predicting text, but that does not automatically make it good at doing what a user asks. Instruction tuning helps bridge that gap. You’d reach for it when you want a model to respond more helpfully to prompts like: “Summarize this email” “Write SQL for this schema” “Explain this code as i

Distillation is a way to train a smaller model to imitate a larger, better one, so you keep much of the quality while reducing cost, latency, or memory use. Distillation solves a very practical problem: the model you want may be too slow, too expensive, or too large to deploy everywhere. You reach for distillation when you want one or more of these: lower inference cost faster responses smaller on-device models simpler deployment a model that approximates a stronger system without retraining fro

Quantization is the process of representing numbers with fewer bits, so a model or computation uses less memory and can often run faster. Large neural networks store weights, activations, and sometimes key/value caches as floating-point numbers. That is accurate, but expensive. Quantization reduces that cost by converting some of those values to lower-precision formats, such as 8-bit integers or even 4-bit representations. You usually reach for quantization when you want one or more of these: lo

An inference endpoint, or serving API, is a network service you send inputs to in order to get model predictions back in real time. It solves the problem of turning a trained model into something applications can actually use. You reach for an inference endpoint when you want: a web or mobile app to call a model on demand a backend to classify, rank, summarize, or generate text without running the model itself a standardized way to deploy, scale, monitor, and secure model requests In practice, m

Batching is the practice of grouping multiple requests together so a model can process them more efficiently, and continuous batching is the version where new requests are added to that group while others are still running. Large models are expensive to run because the same GPU is often underused if you serve one request at a time. Batching improves throughput by sharing compute across many requests. Continuous batching goes a step further: it keeps the GPU busy by letting the server dynamically

Streaming, or token streaming, is when an AI model sends you its output incrementally as it is generated instead of waiting until the whole answer is finished. This solves the “dead air” problem: users see text quickly, which makes apps feel faster and more interactive. It is especially useful for chatbots, copilots, and any UI where a short time-to-first-token matters more than waiting for the full response. In practice, teams use streaming when they want: faster perceived latency, a typing-lik

Time to first token (TTFT) is the time between sending a generative AI request and receiving the model’s first output token. TTFT is the part of latency users feel first. If it is high, the system can seem “stuck” even if the full answer eventually streams quickly. You usually care about TTFT when you want: a responsive chat or copilot experience good perceived performance in streaming UIs to compare serving setups, models, or prompts to separate “slow to start” from “slow to finish” In practice

Tokens per second, or throughput, is the rate at which a language model generates or processes tokens over time. Throughput is a practical way to describe model speed. If you care about user experience, cost efficiency, or server capacity, tokens per second tells you how much text the system can handle in a given time. You’d use it when comparing models, sizing infrastructure, or setting expectations for streaming responses. In practice, teams often care about both throughput and latency

Speculative decoding is a text-generation trick that lets a fast “draft” model propose several next tokens and then uses a larger “target” model to verify them, often reducing end-to-end latency without changing the final output distribution. Autoregressive LLMs normally generate one token at a time, and each token requires a full forward pass through the big model. That is accurate, but it can be slow. Speculative decoding matters when you want lower latency or higher throughput without switchi

Goal-directed behavior in agents is behavior where an agent chooses actions to achieve a specific objective, rather than just reacting passively to the latest input. Goal-directed behavior is the core of what makes an “agent” useful for real work. If a system can keep a goal in view, it can plan, prioritize, and recover from interruptions instead of only producing one-off responses. You reach for this idea when you want a system to: complete multi-step tasks, make tradeoffs between possible acti