A demo that looks great in five hand-picked prompts can fail one in ten in production β and you'd never know. Evaluation and observability are how you turn "seems good" into measured, monitored, and defensible. This is the practice of proving an LLM app works and keeping it working.
The core problem. LLM outputs are open-ended and non-deterministic, so there's rarely one "correct" string to diff against. You need graded judgement, representative test sets, and live monitoring β not just unit tests.
Why Why LLM evaluation is hard
No single answer
Open-ended
Many valid outputs. Exact-match fails; you need graded or semantic judgement.
Non-deterministic
Varies per call
Same input β different output. One good run doesn't prove reliability.
Silent regressions
Prompt/model drift
A prompt tweak or model upgrade can quietly break cases that used to work.
Multi-step
Agents compound
Errors accumulate across tool calls and turns; you must trace the whole chain.
Mindset: treat evaluation as a dataset problem, not a vibe check. If you can't point to a scored test set, you can't claim it works β and you can't safely change anything.
Foundation Build an eval dataset
Everything starts with a curated set of representative inputs β ideally with expected outputs or grading criteria. Grow it from real traffic and every bug you find.
ποΈ What goes in it
Include Happy-path cases, known-hard cases, edge cases, and every past failure (turn each bug into a permanent test β this is your regression suite).
Avoid Only cherry-picked easy prompts; letting it go stale; leaking the eval set into your few-shot prompt (then you're testing memorisation).
Start tiny, stay honest. 20β50 well-chosen, hand-labelled cases beat 10,000 noisy ones. Add every production failure the moment you find it.
Pre-ship Offline evaluation
Run your app over the dataset before shipping a change, and compare scores against the previous version. This is the LLM equivalent of a test suite in CI.
Grader
How it scores
Good for
Exact / rule
String match, regex, JSON-schema valid, code runs
Structured or verifiable outputs
Reference-based
Similarity to a gold answer (embedding / F1 / ROUGE)
Q&A, summarisation with a known answer
LLM-as-judge
A model grades against a rubric
Open-ended quality, tone, helpfulness
Human
People rate or label
Ground truth & calibrating the judge
# Offline eval loop β the shape of every eval harnessfor case in dataset:
output = app(case.input)
case.score = grade(output, case.expected) # rule / reference / judge
pass_rate = mean(c.score for c in dataset)
# Gate the deploy: fail CI if pass_rate drops vs the baselineassert pass_rate >= baseline - tolerance
Technique LLM-as-judge
For open-ended outputs, use a (usually strong) model to grade against an explicit rubric. Powerful and scalable β but only trustworthy if you validate the judge against human labels first.
JUDGE_PROMPT = """You are grading a support answer.Rubric (score 1-5): 5 = correct, complete, cites the policy 3 = correct but missing a detail 1 = wrong or hallucinatedQuestion: {q}Answer: {a}Return JSON: {"score": , "reason": ""}"""
Make the judge reliable: give a concrete rubric & scale, ask for a reason (not just a number), prefer pairwise "A vs B" over absolute scores when possible, watch for bias (position, verbosity, self-preference), and measure the judge's agreement with humans before you trust it.
Post-ship Online evaluation & observability
Offline tells you a change is safe; observability tells you what's actually happening in production. You want to see, sample, and score real traffic.
π Tracing
Capture Every request: full prompt, retrieved context, tool calls, model + params, output, latency, tokens, cost. For agents, trace the whole multi-step chain (spans) so you can see where it went wrong.
π Online signals
Watch Implicit feedback (retries, edits, thumbs, abandonment), sampled LLM-as-judge scores on live traffic, guardrail hits, and drift in latency/cost/refusal rate. Alert on regressions.
π‘οΈ Guardrails
Enforce Input/output checks that run in real time: PII / safety filters, schema & grounding validation, refusal of out-of-scope asks. A failed guardrail is both a block and an eval signal. See AI security.
Metrics What to measure
Dimension
Example metrics
Quality
Accuracy / pass-rate, rubric score, groundedness (is it supported by context?), relevance
For RAG, split the metric: retrieval quality (did we fetch the right chunks? β recall / MRR) vs generation quality (given those chunks, was the answer grounded & correct?). A bad answer from good context is a generation bug; a bad answer from bad context is a retrieval bug. See RAG.
Practice The eval loop
Evaluation isn't a one-off gate β it's a flywheel that ties directly to the agent loop and your release process:
Curate a small, representative dataset with grading criteria.
Grade offline (rules + reference + judge) and set a baseline.
Change one thing β prompt, model, retrieval, tool.
Compare scores vs baseline; gate the deploy in CI.
Observe production via tracing + sampled online evals.
Feed back every failure into the dataset β go to step 2.
Golden rule: change one variable at a time and re-score. If you tweak the prompt and swap the model at once, a score change tells you nothing about which one caused it.
Cheat sheet
Question
Short answer
Why not just unit tests?
Outputs are open-ended & non-deterministic β you need graded evals, not exact match.
Where do I start?
A small, representative, hand-labelled dataset. Grow it from real failures.
Offline vs online?
Offline gates deploys (pre-ship); online monitors real traffic (post-ship).
Grade open-ended output?
LLM-as-judge with a rubric β but validate it against humans first.