#llm-fundamentals

Top-p sampling, also called nucleus sampling, is a way to make a language model pick the next token from only the smallest set of likely options whose combined probability reaches a chosen threshold. If you always choose the single most likely next token, model outputs can become repetitive and brittle. If you sample from all tokens, outputs can get noisy and incoherent. Top-p gives you a middle ground: it keeps the model flexible, but trims away the long tail of very unlikely tokens before samp

Temperature is a sampling setting that controls how random or deterministic an LLM’s next-word choices are when it generates text. Temperature is one of the simplest ways to trade off consistency versus variety. Use a lower temperature when you want stable, repeatable answers: classification-like outputs, structured data, summaries, or code. Use a higher temperature when you want more diversity: brainstorming, creative writing, alternate phrasings, or exploratory chat. In practice, many

Attention, or self-attention, is a way for a model to decide which other words or tokens in the same input matter most when interpreting each token. Self-attention solves a basic language problem: the meaning of a word often depends on other words nearby or far away. It lets a model build a context-aware representation of each token instead of treating tokens mostly independently or relying only on fixed-size windows. In practice, this is why Transformers work so well for text, code, and other s

A transformer is a neural network architecture that processes sequences by using attention to compare each token with the others, making it a flexible foundation for modern language models and many other sequence tasks. Before transformers, many sequence models had to read text step by step, which made long-range dependencies harder to handle and training slower to parallelize. Transformers solve this by letting the model look at many positions in the input at once. In practice, you reach for a

A token is a chunk of text that a language model reads and predicts, and tokenization is the process of splitting text into those chunks. LLMs do not operate directly on raw characters or words; they operate on tokens. That matters because: Input limits are measured in tokens, not characters. Cost and latency are often tied to token count in LLM APIs. Model quality depends on how text is tokenized: the same sentence can become very different token sequences across tokenizers.

A mixture-of-experts (MoE) model is a neural network that routes each input, or part of an input, to only a small subset of specialized submodels (“experts”) instead of using the whole model every time. MoE models are useful when you want much larger capacity without paying the full compute cost on every token or example. In practice, that means you can scale model size to improve quality while keeping inference and training more efficient than a dense model of similar total parameter count.