Skip to main content
Back to module
Applied Machine Learning · Working With LLMs

Evaluating models and LLM apps

Reading50mLesson 3 of 3 in this module

Build a fixed evaluation set with references and tags, and score a model or LLM app with exact match and token F1.

Why this matters

"It looked good when I tried it" is not evaluation. A model or LLM app that seemed fine on five questions can fail on a whole category of real inputs, and a prompt tweak that fixes one case can quietly break ten others. A fixed evaluation set with an automatic score lets you compare models, prompts and settings honestly, catch regressions before users do, and say with evidence whether a change helped. Building one is often the highest-value work on an ML project.

The core idea: a fixed, representative test set plus a metric

An evaluation set is a list of cases. Each case has an input, a reference (the expected answer, label, or the source that should be retrieved) and tags that describe it:

{"id": "q4", "question": "Can I return a gift card?",
 "reference": "No, gift cards are final sale", "tags": ["returns"]}

You run the system on every input, score each output against its reference, and report the average overall and per tag. The per-tag view is where you learn things: "returns questions score 0.57, payments 0.95" tells you where to look.

Where cases come from

  • Real traffic (anonymised logs, past tickets): the best source, because it reflects what users actually ask.
  • Domain experts writing questions and gold answers.
  • Known failures: every bug report becomes a test case so it never comes back.
  • Edge cases on purpose: typos and slang, very long inputs, questions the system should refuse or answer with "I don't know", other languages, adversarial inputs such as prompt injection.
  • Synthetic cases generated by an LLM can add coverage, but review them by hand; they tend to be easier than real ones.

Aim for at least 50–100 cases to start, cover every category you care about, and keep the set frozen and versioned. If you tune prompts against it for weeks, you are overfitting to it, so keep a separate held-out set for the final check, just as with model selection.

Metrics for different outputs

  • Classification (labels): accuracy, precision, recall, F1 per class (Machine Learning Foundations).
  • Retrieval: recall@k, the share of questions whose correct source is in the top k.
  • Short factual answers: exact match after normalisation, and token F1, the overlap between predicted and reference words.
  • Open-ended text (summaries, explanations): human rating with a clear rubric, or LLM-as-a-judge, where a strong model grades each answer against the reference and rubric. Judges are useful at scale but have biases (they favour longer answers, and their own style), so check a sample of their grades against human ones.
  • Operational: latency, cost per request, refusal rate, and format validity (did the JSON parse?).

Worked example: token F1

Normalise both strings (lowercase, strip punctuation and the articles "a", "an", "the"), split into tokens, and count the shared tokens:

prediction: "the cat sat"      -> [cat, sat]
reference:  "a cat sat down"   -> [cat, sat, down]
shared = 2
precision = 2/2 = 1.0    recall = 2/3 = 0.67
F1 = 2 * 1.0 * 0.67 / (1.0 + 0.67) = 0.8

Exact match would score this 0. Token F1 gives partial credit, but it also has blind spots: "Usually 3-5 business days" against "3 to 5 business days" scores only 0.44 although both mean the same thing. That is why you read the failures: a metric tells you where to look, not what is true.

from collections import Counter

def token_f1(pred_tokens, ref_tokens):
    common = sum((Counter(pred_tokens) & Counter(ref_tokens)).values())
    if common == 0:
        return 0.0
    p, r = common / len(pred_tokens), common / len(ref_tokens)
    return 2 * p * r / (p + r)

Tools

Simple evaluation needs only a CSV or JSON file and a loop like the one in the activity. As projects grow, libraries help: scikit-learn metrics for classic models, Hugging Face evaluate, and open-source LLM evaluation tools such as Ragas (for RAG), promptfoo or DeepEval. Experiment trackers (next module) store each evaluation run so you can compare them. Run the evaluation in CI so every prompt or model change is scored automatically.

Common mistakes

  • Too few or too easy cases, or all from one category.
  • Changing the evaluation set between the runs you compare.
  • One headline number only: always break down by tag and read the worst cases.
  • Trusting an LLM judge blindly without checking it against human grades.
  • Testing at a high temperature with one run: results vary, so use low temperature or average several runs.
  • Personal data in the set: anonymise real logs before they become test fixtures.

Quick recap

  • An evaluation set is a frozen, versioned list of inputs, references and tags.
  • Pick metrics that fit the output: classification metrics, recall@k, exact match and token F1, rubric or judge scores, plus latency and cost.
  • Report overall and per tag, and read the failures by hand.
  • Every bug becomes a test case; keep a held-out set to avoid overfitting your prompts.

Try it

Open the Do tab. You will write normalisation, exact match, token F1 and an evaluation report in plain Python, add edge cases to a help-desk evaluation set, and catch a wrong answer automatically.

Also in this lesson: