Evaluation & Testing

Part 1 · Evaluation

The four-layer evaluation framework

A good way to organize checks is in four layers, ordered from cheapest and shallowest to most expensive and deepest. The rule is simple: run a layer only if the layers before it passed. If the output is not even valid, there is no point paying a language model to grade it. This "stop as soon as something fails" ordering keeps most checks free.

Before we go through the layers, one distinction that decides when each check runs. Some checks run before you ship, on a fixed set of test examples, in your build pipeline. This is called offline evaluation. Other checks run on live user traffic after you ship, watching real requests as they happen. This is called online evaluation. Layers 1 to 3 are usually run offline on test examples; Layer 4 and lighter versions of the others also run online. Part 6 covers online monitoring in detail.

Part 2 · Evaluation

Reference & semantic matching metrics

Layer 2 is used when you already have a correct answer to compare against, called the reference or gold answer. The goal is to check whether the generated text means the same thing as the reference, not whether the exact characters match. There are three families of methods, in rising order of sophistication.

2.1 Counting shared words: BLEU and ROUGE

The oldest methods just count how many short runs of words the output and the reference have in common. A run of n consecutive words is called an n-gram: "the cat" is a 2-gram, "the big cat" is a 3-gram.

The recall-focused metric, which asks "how much of the reference did the output cover?", is called ROUGE (Recall-Oriented Understudy for Gisting Evaluation). The precision-focused metric, which asks "how much of the output was actually justified by the reference?", is called BLEU (Bilingual Evaluation Understudy). Here is a tiny worked example.

2.2 Comparing meaning with embeddings

A better approach turns each piece of text into a list of numbers that captures its meaning. That list of numbers is called an embedding, and the model that produces it is an embedding model. Two texts that mean similar things get embeddings that point in a similar direction. We measure how similar two directions are with the cosine similarity: the cosine of the angle between the two lists of numbers. It runs from 1 (same direction, same meaning) down toward 0 (unrelated).

Here \(\mathbf v_g\) is the generated answer's embedding and \(\mathbf v_r\) is the reference's. The top is the dot product (multiply matching numbers, add them up). The bottom divides by each list's length so only direction matters, not size.

# minimal embedding-similarity check
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")
gen  = "The capital of France is Paris."
ref  = "Paris is the capital of France."

v_g, v_r = model.encode([gen, ref])
score = util.cos_sim(v_g, v_r).item()   # ~0.94
assert score > 0.82   # passes: same meaning, different word order

2.3 BERTScore — matching word by word, in context

The previous method squeezes a whole sentence into one embedding, which can blur details. A refinement embeds every word in its context, then matches each output word to the reference word it is most similar to, and averages those similarities. This method is called BERTScore (it uses a model called BERT, Bidirectional Encoder Representations from Transformers). Because it compares word by word in context, it lines up with human judgment better than BLEU or ROUGE on many tasks.

Part 3 · Evaluation

LLM-as-a-judge — power and pitfalls

The deepest and most flexible check is to hand the output to another language model and ask it to grade the answer against a set of instructions. This is called LLM-as-a-judge (LLM stands for Large Language Model). It can rate things no formula can, such as accuracy, tone, helpfulness, or whether the answer stuck to a source. It is also the easiest to trust too much, because these judges have consistent, predictable biases that you have to actively cancel out.

3.1 The two ways to ask a judge

• Give one answer a score: "Rate this answer from 1 to 5 for factual accuracy." Simple, but the scores tend to drift over time and bunch up in the middle. This is called absolute scoring.
• Compare two answers: "Here are answers A and B. Which is better?" More reliable, because deciding which of two things is better is a steadier judgment than putting a number on one thing alone. This is called pairwise comparison, and it is the basis of the A/B regression testing in Part 6.

Here is a concrete judge prompt for the pairwise case.

SYSTEM: You are a strict grader. You will see a QUESTION, a
REFERENCE answer, and two candidate answers A and B. Decide
which candidate better matches the reference on factual accuracy.
Ignore length and writing style. Do not reward extra detail that
is not in the reference. Reply with exactly one token: A, B, or TIE.

QUESTION: What is the capital of France?
REFERENCE: Paris.
A: The capital of France is Paris.
B: France is a country in Western Europe with many large cities.

ASSISTANT: A

3.2 G-Eval — a more reliable recipe

A widely used improvement on plain judging is called G-Eval. It adds two ideas. First, before grading, it asks the judge to write out its own step-by-step checklist from the criteria, so the grading is more consistent. Second, instead of taking the single score the model happened to output, it looks at how sure the model was about each possible score and takes a weighted average. The model's confidence in each score is called its probability for that score, and turning raw model outputs into probabilities that add up to 1 is done with the softmax function.

3.3 The three biases you must correct

A "bias" here means a consistent way the judge is unfair that has nothing to do with actual answer quality. There are three you should assume are present.

3.4 RAG-specific model-based metrics

Many systems answer a question by first fetching relevant documents and then writing an answer using them. This is called RAG (Retrieval-Augmented Generation), and Part 5 covers it fully. The fetched documents are called the retrieved context. Three model-based checks are the standard way to grade a RAG system; together they are often called the "RAG triad" and are implemented by tools named Ragas and TruLens. Each works by having a language model break the text into small factual statements, called claims, and checking each one.

• Faithfulness (also called groundedness): the fraction of the answer's claims that the retrieved documents actually support. This is the main signal for catching made-up content, called hallucination.
• Answer relevancy: does the answer actually address the question? Ragas estimates this by asking a model to invent questions the answer would fit, then measuring the cosine similarity between those invented questions and the real one.
• Context precision and recall: did the retrieval step fetch the right documents? Precision asks whether the fetched documents were relevant and well-ranked; recall asks whether it fetched all the relevant ones. These separate a retrieval problem from a writing problem.

Part 4 · Evaluation

Golden datasets — the asset that compounds

None of the metrics above mean anything without a set of examples to run them on. That curated set of test cases — each one an input paired with its expected output and the way it should be graded — is called a golden dataset. It is the most valuable and longest-lived asset in an LLMOps project. Models change, prompts change, vendors change; the golden dataset is the fixed ruler you measure all of them against.

4.1 What goes into it

• Common requests (happy paths): the everyday, typical inputs. Usually the smallest and least interesting part.
• Edge cases: very long inputs, empty inputs, vague requests, other languages, and inputs written to trip the system up.
• Regression cases: every past production failure, frozen forever as a permanent test. A "regression" is when a change re-breaks something that used to work; this is the feedback loop from Part 1 made concrete.
• A grading rule per case: exact match, a keyword that must appear, a cosine-similarity cutoff, or a judge checklist. The grading method is stored alongside each example.

Part 5 · Evaluation

Testing the LLM layers of an agentic app

Evaluating a plain model asks "is the text good?" But many real applications do more than write text. An agent is a system that uses a model to decide which actions to take, calling tools, querying APIs or MCP servers (MCP means Model Context Protocol, a standard way to expose tools to a model), and carrying out multi-step plans. Testing an agent asks more questions: did it pick the right tool, pass valid inputs to it, take a sensible sequence of steps, and follow the rules of its domain?

The standard way to organize these tests is a testing pyramid: many fast cheap tests at the bottom, fewer slow expensive tests at the top.

5.1 Tier 1 — plain unit tests

Because the output follows a fixed format (from Part 1), the tool inputs and the final output are typed, so an ordinary assertion checks them with no model call at all.

def test_listing_structure_and_constraints():
    out = run_listing_agent("Men's running shoe, blue, lightweight, size 10")
    assert len(out.title) <= 80          # marketplace hard limit
    assert len(out.bullet_points) >= 3    # minimum feature bullets
    assert out.category in ALLOWED_TAXONOMY

5.2 Tier 2 — path and tool tests

The sequence of steps an agent takes is called its trajectory. An agent can reach a right-looking answer by a wrong path. To test this cheaply and repeatably, you replace the real tools with stand-ins that return fixed responses, called mocks, then check the exact list of calls the agent made.

def test_mcp_tool_trajectory():
    tracer = AgentTracer()
    run_listing_agent("Men's running shoe", tracer=tracer)
    # Correct grounding: look up the taxonomy BEFORE writing copy
    assert "mcp_get_category_taxonomy" in tracer.tool_calls
    assert tracer.index("mcp_get_category_taxonomy") < tracer.index("generate_copy")
    assert tracer.tool_call_count < 5      # no runaway tool loop

This catches two failures that an output-only test would miss: the agent skipping a required lookup step and then guessing the category, and the agent getting stuck in a loop that burns time and money.

5.3 Tier 3 — domain evaluation, with two worked case studies

Case A · E-commerce product-listing agent

The top-tier test checks that the agent did not invent product features that were not in the supplier's text (a hallucination).

def test_no_hallucinated_attributes():
    score = evaluate_hallucination(
        input="Lightweight mesh canvas shoe",
        output=generated_listing,
        context="Mesh canvas is not waterproof.")
    assert score < 0.1   # agent must not claim 'waterproof'

Case B · Medical literature-synthesis agent

Here the stakes make Layers 1 and 2 not enough on their own; the ways it can fail are specific to the domain and unsafe.

Part 6 · Evaluation

CI/CD for prompts & agents

All of the above only helps if it runs automatically on every change. Running your tests automatically whenever code changes is called CI/CD (Continuous Integration and Continuous Delivery). The pattern is the same as ordinary software CI/CD, with two extra twists for language models: you must control cost, and you compare the new version's scores against the current production version rather than just checking pass/fail.

6.1 The tool landscape

Here is a minimal example of one of these frameworks, DeepEval, which lets you write evals in the same style as ordinary unit tests.

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import GEval

correctness = GEval(
    name="Correctness",
    criteria="Is the answer factually correct vs the reference?",
    threshold=0.7,   # fail the test below this score
)

def test_capital_question():
    case = LLMTestCase(
        input="What is the capital of France?",
        actual_output="The capital of France is Paris.",
        expected_output="Paris.")
    assert_test(case, [correctness])   # runs the judge, asserts score >= 0.7