Author

aiterms

What is RAG evaluation (faithfulness and relevance)?

RAG evaluation is the process of checking whether a retrieval-augmented generation system’s answer is faithful to the provided context and relevant to the user’s question.

Why it matters

RAG systems are meant to answer with help from retrieved documents, not just from the model’s memory. Evaluation tells you whether the system is actually doing that well.

In practice, teams use RAG evaluation to catch two common failure modes:

• Low faithfulness: the answer sounds plausible but is not supported by the retrieved evidence.
• Low relevance: the answer may be factually grounded, but it does not address the user’s question well.

If you only measure one of these, you can miss serious issues. A response can be relevant but hallucinated, or faithful to the context but still unhelpful.

How it works

A typical RAG evaluation splits the quality check into two parts:

1. Faithfulness

   • Ask: Is every important claim in the answer supported by the retrieved context?
   • This is a grounding check. If the answer says something the documents do not support, faithfulness is low.

2. Relevance

   • Ask: Does the answer actually address the user’s query?
   • This checks whether the system stayed on task and answered the right thing, rather than drifting into related but unnecessary details.

3. Sometimes evaluate at multiple levels

   • Some setups score the retrieved passages themselves for relevance to the query.
   • Others score the final answer for both faithfulness and relevance.
   • These are related but not identical. Good retrieval helps relevance, but it does not guarantee a faithful answer.

4. Use human review, automatic judges, or both

   • Human evaluation is the clearest standard when stakes are high.
   • Automated evaluation often uses an LLM-as-judge or heuristic scoring to scale coverage.
   • The exact rubric matters: different papers and tools define these metrics slightly differently, so teams should read the rubric carefully rather than assume a universal definition.

Tiny concrete example

User question: “What year was the company founded?”

Retrieved context: “The company was founded in 2014 in Berlin.”

Answer A: “The company was founded in 2014.”

• Faithful: yes
• Relevant: yes

Answer B: “The company was founded in 2014, and its headquarters are in Paris.”

• Faithful: partly no, if Paris is not in the context
• Relevant: yes

Answer C: “The company uses a subscription model.”

• Faithful: maybe, if the context says so
• Relevant: no, because it does not answer the founding-year question

Common pitfalls / when NOT to use it

• Confusing retrieval quality with answer quality
  A bad answer can come from bad retrieval, bad generation, or both. Evaluate them separately when possible.

• Treating “faithful” as “true in the world”
  Faithfulness usually means “supported by the provided context,” not necessarily globally correct.

• Using a vague rubric
  If you do not specify what counts as support, judges may score inconsistently.

• Over-trusting automatic metrics
  LLM judges are useful, but they can be noisy, biased, or overly lenient. Spot-check with humans.

• Using only aggregate scores
  A single average can hide systematic failures on certain question types, document types, or languages.

• Skipping evaluation on the retrieval side
  If the retriever misses the right evidence, the generator may look “unfaithful” even though the root cause is retrieval.

Related terms

• What is retrieval-augmented generation (RAG)?
• What is a vector database?
• What is an embedding?
• What is semantic search?
• What is chunking (and semantic chunking)?
• What is hybrid search (BM25 + vector)?
• What is a reranker / cross-encoder?
• What is top-k retrieval?
• What is a knowledge base for RAG?
• What is contextual / late chunking?
• What is RAG vs fine-tuning?
• What is a vector index (HNSW / IVF)?
• What is cosine similarity?
• What is metadata filtering in retrieval?
• What is query rewriting / expansion?
• What is the lost-in-the-middle problem?
• What is graph RAG?
• What is a document loader / parser?

同じ著者の記事

What is graph RAG?

Graph RAG, short for Graph Retrieval-Augmented Generation, is a way to help an LLM answer questions by retrieving information from a graph of connected entities and relationships instead of, or in addition to, plain text chunks. Classic RAG works well when the answer lives in a few relevant text passages. But some questions are really about relationships: who depends on whom, how events connect, which product is linked to which system, or how a concept appears across many documents.

papoo.work

What is the lost-in-the-middle problem?

The lost-in-the-middle problem is the tendency of a language model to miss information that appears in the middle of a long context, even when that information is relevant to the answer. This matters any time you stuff a lot of text into an LLM prompt: long documents, chat histories, codebases, retrieval-augmented generation (RAG), or multi-step agent traces. In practice, it means: the model may answer well using the beginning or end of the context, but fail on facts buried in the middle, so

papoo.work

What is query rewriting / expansion?

Query rewriting or query expansion is the process of changing a user’s search query into a better one—usually by adding, removing, or rephrasing terms so a search or retrieval system can find more relevant results. People often search with short, vague, or underspecified queries. Systems that use the raw query alone may miss relevant documents because the wording does not match the target content. Query rewriting/expansion helps when you want better recall, better intent matching, or more robust

papoo.work

What is metadata filtering in retrieval?

Metadata filtering in retrieval is the practice of narrowing a search or vector search to documents that match structured fields like date, author, source, tenant, language, or content type before or during ranking. It solves a very practical problem: not every relevant item is relevant for this user, this time, or this task. If you are building search or retrieval-augmented generation (RAG), metadata filters help you: exclude clearly irrelevant documents early, enforce access control or tenant

papoo.work

What is cosine similarity?

Cosine similarity is a way to measure how similar two vectors are by comparing the angle between them, not their size. It’s useful when you care more about direction than magnitude. In AI and information retrieval, that usually means comparing embeddings, text vectors, or feature vectors to find items that “mean the same thing” even if they have different lengths. You’d reach for cosine similarity when: ranking search results by semantic closeness, comparing document, sentence, or image embe

papoo.work

What is a vector index (HNSW / IVF)?

A vector index is a data structure that helps a system quickly find the most similar embeddings or vectors to a query vector; it is also commonly called an ANN index, short for approximate nearest neighbor index. If you store thousands, millions, or billions of vectors and want “find the closest ones,” a brute-force scan is usually too slow. A vector index lets you trade a little exactness for much faster retrieval, which is why it shows up in search, retrieval-augmented generation (RAG), recomm

papoo.work

What is contextual / late chunking?

Contextual chunking, also called late chunking, is a way to split a document into retrieval chunks after a model has already encoded the whole document, so each chunk keeps some awareness of the surrounding context. Normal chunking cuts text first and then embeds each piece alone. That is simple, but it can make chunks semantically thin: a paragraph may be ambiguous without the surrounding sections, headings, or earlier definitions. Late chunking helps when you want better retrieval quality

papoo.work

What is a knowledge base for RAG?

A knowledge base for RAG is the collection of documents or records a system retrieves from to ground an LLM’s answer in relevant source material. RAG, short for retrieval-augmented generation, works best when the model can search a useful, trustworthy set of content before it writes. That content is the knowledge base. You’d reach for one when the model needs to answer questions from: company docs support articles policies and handbooks product specs research notes tickets or tickets-like record

papoo.work

What is top-k retrieval?

Top-k retrieval is a way to search a collection and return the k most relevant items for a query, instead of trying to sort or inspect everything. Top-k retrieval is the standard first step in search, recommendation, and retrieval-augmented generation (RAG). You use it when you want a fast shortlist: the most relevant documents, passages, products, images, or candidates, without paying the cost of full ranking over the entire corpus. In practice, teams reach for top-k retrieval when: the dat

papoo.work

What is a reranker / cross-encoder?

A reranker, often implemented as a cross-encoder, is a model that takes a query and a candidate result together and scores how well they match. A reranker solves the “good enough first pass, better final answer” problem. In search and retrieval systems, you usually start with a fast retriever that finds a few dozen or few hundred candidate documents. That first stage is optimized for speed, not perfect ranking. A reranker then looks at those candidates more carefully and reorders them so the mos

papoo.work