← All tutorials
LLM Eval & Observability

LLM evaluation & observability

A demo that looks great in five hand-picked prompts can fail one in ten in production β€” and you'd never know. Evaluation and observability are how you turn "seems good" into measured, monitored, and defensible. This is the practice of proving an LLM app works and keeping it working.

The core problem. LLM outputs are open-ended and non-deterministic, so there's rarely one "correct" string to diff against. You need graded judgement, representative test sets, and live monitoring β€” not just unit tests.

Why  Why LLM evaluation is hard

No single answer

Open-ended

Many valid outputs. Exact-match fails; you need graded or semantic judgement.

Non-deterministic

Varies per call

Same input β†’ different output. One good run doesn't prove reliability.

Silent regressions

Prompt/model drift

A prompt tweak or model upgrade can quietly break cases that used to work.

Multi-step

Agents compound

Errors accumulate across tool calls and turns; you must trace the whole chain.

Mindset: treat evaluation as a dataset problem, not a vibe check. If you can't point to a scored test set, you can't claim it works β€” and you can't safely change anything.

Foundation  Build an eval dataset

Everything starts with a curated set of representative inputs β€” ideally with expected outputs or grading criteria. Grow it from real traffic and every bug you find.

πŸ—‚οΈ What goes in it

Include Happy-path cases, known-hard cases, edge cases, and every past failure (turn each bug into a permanent test β€” this is your regression suite).

Avoid Only cherry-picked easy prompts; letting it go stale; leaking the eval set into your few-shot prompt (then you're testing memorisation).

Start tiny, stay honest. 20–50 well-chosen, hand-labelled cases beat 10,000 noisy ones. Add every production failure the moment you find it.

Pre-ship  Offline evaluation

Run your app over the dataset before shipping a change, and compare scores against the previous version. This is the LLM equivalent of a test suite in CI.

GraderHow it scoresGood for
Exact / ruleString match, regex, JSON-schema valid, code runsStructured or verifiable outputs
Reference-basedSimilarity to a gold answer (embedding / F1 / ROUGE)Q&A, summarisation with a known answer
LLM-as-judgeA model grades against a rubricOpen-ended quality, tone, helpfulness
HumanPeople rate or labelGround truth & calibrating the judge
# Offline eval loop β€” the shape of every eval harness
for case in dataset:
    output = app(case.input)
    case.score = grade(output, case.expected)   # rule / reference / judge

pass_rate = mean(c.score for c in dataset)
# Gate the deploy: fail CI if pass_rate drops vs the baseline
assert pass_rate >= baseline - tolerance

Technique  LLM-as-judge

For open-ended outputs, use a (usually strong) model to grade against an explicit rubric. Powerful and scalable β€” but only trustworthy if you validate the judge against human labels first.

JUDGE_PROMPT = """You are grading a support answer.
Rubric (score 1-5):
  5 = correct, complete, cites the policy
  3 = correct but missing a detail
  1 = wrong or hallucinated

Question: {q}
Answer:   {a}
Return JSON: {"score": , "reason": ""}"""
Make the judge reliable: give a concrete rubric & scale, ask for a reason (not just a number), prefer pairwise "A vs B" over absolute scores when possible, watch for bias (position, verbosity, self-preference), and measure the judge's agreement with humans before you trust it.

Post-ship  Online evaluation & observability

Offline tells you a change is safe; observability tells you what's actually happening in production. You want to see, sample, and score real traffic.

πŸ”­ Tracing

Capture Every request: full prompt, retrieved context, tool calls, model + params, output, latency, tokens, cost. For agents, trace the whole multi-step chain (spans) so you can see where it went wrong.

πŸ“Š Online signals

Watch Implicit feedback (retries, edits, thumbs, abandonment), sampled LLM-as-judge scores on live traffic, guardrail hits, and drift in latency/cost/refusal rate. Alert on regressions.

πŸ›‘οΈ Guardrails

Enforce Input/output checks that run in real time: PII / safety filters, schema & grounding validation, refusal of out-of-scope asks. A failed guardrail is both a block and an eval signal. See AI security.

The evaluation and observability loop Users hit the LLM app; every call is traced; traces feed observability and get sampled into the eval dataset; offline evals gate the next deploy β€” a closed loop. Usersreal traffic LLM app+ guardrails Trace storeprompts Β· cost Β· spans Observabilitydashboards Β· alerts sample failures β†’ eval dataset Offline evals gate deploy

Metrics  What to measure

DimensionExample metrics
QualityAccuracy / pass-rate, rubric score, groundedness (is it supported by context?), relevance
SafetyHallucination rate, toxicity, PII leakage, jailbreak / refusal rate
TaskTask success, tool-call correctness, format/schema validity, citation accuracy
OpsLatency (p50/p95), tokens & cost per request, throughput, error rate
UserThumbs up/down, edit/retry rate, abandonment, CSAT
For RAG, split the metric: retrieval quality (did we fetch the right chunks? β€” recall / MRR) vs generation quality (given those chunks, was the answer grounded & correct?). A bad answer from good context is a generation bug; a bad answer from bad context is a retrieval bug. See RAG.

Practice  The eval loop

Evaluation isn't a one-off gate β€” it's a flywheel that ties directly to the agent loop and your release process:

  1. Curate a small, representative dataset with grading criteria.
  2. Grade offline (rules + reference + judge) and set a baseline.
  3. Change one thing β€” prompt, model, retrieval, tool.
  4. Compare scores vs baseline; gate the deploy in CI.
  5. Observe production via tracing + sampled online evals.
  6. Feed back every failure into the dataset β†’ go to step 2.
Golden rule: change one variable at a time and re-score. If you tweak the prompt and swap the model at once, a score change tells you nothing about which one caused it.

Cheat sheet

QuestionShort answer
Why not just unit tests?Outputs are open-ended & non-deterministic β€” you need graded evals, not exact match.
Where do I start?A small, representative, hand-labelled dataset. Grow it from real failures.
Offline vs online?Offline gates deploys (pre-ship); online monitors real traffic (post-ship).
Grade open-ended output?LLM-as-judge with a rubric β€” but validate it against humans first.
Observability = ?Trace every call (prompt, context, tools, cost, latency) + dashboards + alerts.
Golden rule?Change one variable at a time, then re-score against the baseline.

Knowledge check

Six questions. Pick an answer for each, then check your score.

1. Why can't you evaluate most LLM apps with exact-match unit tests?

2. What's the best first step in an eval practice?

3. Offline evaluation is mainly used to…

4. Before trusting LLM-as-judge scores, you should…

5. In production, observability primarily means…

6. You changed the prompt and swapped the model at the same time; the score moved. What went wrong?

Next steps

🎚️ Fine-tune · RAG · prompt

Choose what to change β€” then evaluate the change.

πŸ” Getting started with loops

The agent loop evals plug into.

πŸ›‘οΈ Web & AI security

Guardrails β€” safety as an eval signal.

πŸ““ All tutorials

The hub β€” LLMs, agents, ML & more.