Go from classic ML to the models used at work: neural networks, embeddings, CNNs, gradient boosting and LLM apps, plus evaluation, experiment tracking and a capstone.

Applied Machine Learning

## Why this matters

Large language models (LLMs) are now part of many data science and ML engineering jobs: summarising documents, extracting fields from messy text, classifying tickets without training data, answering questions over company documents. To use them well you need a working model of what they actually do. Most surprises (a made-up citation, a different answer each time, a forgotten instruction, a surprising bill) follow directly from three ideas: **tokens**, the **context window** and **sampling**.

## The core idea: predict the next token

An LLM is a neural network trained on a huge amount of text to do one thing: given a sequence of tokens, output a probability for every possible **next token**. To generate text it repeats that step: pick a token, append it to the input, predict again.

```text
input:  "The capital of France is"
model:  " Paris" 0.92   " a" 0.03   " the" 0.02   ...
append " Paris", predict again: "." 0.71   "," 0.18 ...
```

Nothing in this loop looks facts up. The model produces text that is likely given its training data and the prompt. That is why it can write fluent, confident and wrong answers (often called **hallucinations**), and why grounding the model in real documents (next lesson) matters.

## Tokens

Models read **tokens**, not words or characters. A tokenizer (usually byte-pair encoding) splits text into frequent chunks. One tokenizer might split a sentence like this (the exact pieces differ between models):

```text
"Tokenization is unbelievable"  ->  ["Token", "ization", " is", " un", "believ", "able"]
```

Rules of thumb for English: one token is about four characters, or about three quarters of a word. Code, numbers and many non-English languages use more tokens per word. Tokens matter because:

- **Pricing and limits** are per token (input and output are often priced differently).
- **Spelling and arithmetic** are hard: the model sees "believ" as one unit, not letters, and numbers may be split oddly.
- Every tokenizer is different, so count with the right one (for example the `tiktoken` library for OpenAI models, or the tokenizer that ships with an open model on Hugging Face).

## The transformer and attention

Modern LLMs are **transformers**. Each token becomes an embedding (as in the text lesson), and a stack of layers updates those vectors. The key operation is **self-attention**: each token computes how relevant every earlier token is and mixes in information from them. That is how "it" in "The trophy didn't fit in the case because it was too big" can draw on "trophy". Attention looks only backwards in a generative model, so each prediction depends on everything before it in the input.

Training happens in stages. **Pretraining** on large text collections gives the next-token skill. **Instruction tuning** and **preference tuning** (for example reinforcement learning from human feedback) teach the model to follow instructions, use a chat format and decline harmful requests.

## The context window

The **context window** is the maximum number of tokens the model can consider at once: the system instructions, the conversation so far, any documents you paste in and the answer it is writing. Windows range from a few thousand to over a million tokens depending on the model. Consequences:

- The model has **no memory** between calls. A chat app resends the conversation every turn.
- Anything outside the window does not exist for the model. Long conversations get truncated or summarised.
- More context costs more and is slower, and models can miss details buried in the middle of very long inputs. Put the important instructions clearly at the start or end, and send only what is relevant.
- The model's built-in knowledge stops at its **training cutoff**. Recent facts must come from the context.

## Sampling: why answers vary

After computing next-token probabilities, the model must choose one. **Sampling settings** control how:

- **Temperature** rescales the probabilities. Near 0, the top token almost always wins (focused and repeatable). Around 1, lower-probability tokens get picked more often (more varied, more creative, more errors).
- **Top-p** (nucleus sampling) samples only from the smallest set of tokens whose probabilities add up to `p`, for example 0.9, which cuts off the long tail of unlikely tokens.
- **Max tokens** caps the length of the output.

Worked example with three candidates:

```text
probabilities at temperature 1:     "Paris" 0.70   "Lyon" 0.20   "banana" 0.10
at a low temperature (0.2):         "Paris" ~0.998 "Lyon" ~0.002 "banana" ~0.00006
with top-p = 0.85 (temperature 1):  keep Paris + Lyon (0.70 + 0.20 >= 0.85), drop banana
```

For extraction, classification and evaluation, use a low temperature. Even at temperature 0, outputs may vary slightly between runs because of hardware and batching, so never assume perfect determinism; test with several runs.

## Using an LLM in an application

- Write a clear **system prompt**: the task, the output format, what to do when unsure ("say you don't know").
- Ask for **structured output** (JSON with named fields) when code will read the answer, and validate it.
- Give **examples** in the prompt (few-shot) for tricky formats.
- **Evaluate** on a fixed set of test inputs before and after every prompt or model change (lesson 3 of this module).
- Treat inputs from users or documents as untrusted: **prompt injection** ("ignore previous instructions...") can arrive inside a document.
- Know where data goes: hosted APIs send your text to a provider. Check your organisation's policy before sending personal or confidential data. Open-weight models (run locally with tools such as Ollama or Hugging Face) keep data in-house.

## Common mistakes

- **Treating output as fact.** Verify claims, or ground the model in sources.
- **Relying on the model for exact arithmetic or counting.** Do the maths in code.
- **High temperature for tasks with one right answer.**
- **Stuffing the whole knowledge base into the prompt** instead of retrieving what is relevant.
- **Changing prompts without re-running an evaluation set.**

## Quick recap

- An LLM predicts the next token from everything in its context, repeatedly.
- Tokens drive cost, limits and odd failures with spelling and numbers.
- The context window is all the model can see; it has no memory between calls and a training cutoff.
- Temperature and top-p trade consistency for variety; use low values for factual tasks.


Explain how large language models predict tokens, what the context window limits, and how sampling settings change outputs.

How LLMs work

Large Language Models explained briefly

How Large Language Models Work

Pick a corpus of 10 to 30 documents (for example a project's docs pages, a public handbook, or your own course notes) and save them as plain text or Markdown files.

Write a one-paragraph design: the users, the questions they ask, the chunk size and overlap you will start with, the metadata you will keep (source, title, date) and how you would enforce access control in a real deployment.

Chunk the documents (start with about 300 words and 50 words of overlap, split on headings or paragraphs) and read 10 random chunks to check the text extraction is clean.

Install sentence-transformers (`pip install sentence-transformers`), embed every chunk with `all-MiniLM-L6-v2` using `normalize_embeddings=True`, and store the vectors in a NumPy array alongside a list of chunk metadata.

Write 15 test questions and, for each, note which source document holds the answer. Include two questions your documents cannot answer.

For each question, embed it, take the dot product with every chunk vector, and list the top 5 chunks with their sources.

Compute recall@5: the share of answerable questions whose correct source appears in the top 5. Record the value.

Change one design choice (chunk size, overlap, or adding the document title to each chunk), re-run, and compare recall@5.

Write the prompt template you would send to an LLM: instructions to answer only from the numbered sources, cite them as [1], [2], and say "I don't know" when the sources do not contain the answer.

## Why this matters

An LLM knows only what was in its training data, up to its cutoff, and it will answer confidently either way. Your organisation's policies, product manuals and last week's tickets are not in there. **Retrieval-augmented generation (RAG)** fixes this without retraining: at question time, find the most relevant passages from your own documents and put them in the prompt, so the model answers from evidence and can cite it. RAG is the most common LLM pattern in industry, and most of its quality comes from ordinary data and search engineering, not from the model.

## The core idea: retrieve, then generate

A RAG system has two phases.

**Indexing (offline, once per document update):**

1. **Load** documents (PDFs, web pages, wiki exports, tickets) and extract clean text.
2. **Chunk** them into passages of a few hundred tokens, with a small overlap so sentences are not cut in half.
3. **Embed** each chunk with an embedding model (the text lesson) and store the vectors plus metadata (source, title, date, permissions) in a **vector index**.

**Answering (online, per question):**

4. **Embed the question** with the same model.
5. **Retrieve** the top-k most similar chunks (cosine similarity), often 3 to 10.
6. **Generate**: build a prompt with instructions, the retrieved chunks and the question, and send it to the LLM.

```text
question --> embed --> vector search --> top-k chunks --+
                                                         +--> prompt --> LLM --> answer + citations
instructions -------------------------------------------+
```

A typical prompt template:

```text
Answer the question using ONLY the sources below. Cite sources as [1], [2].
If the sources do not contain the answer, say "I don't know".

[1] (returns-policy.md) Items can be returned within 30 days of delivery...
[2] (gift-cards.md) Gift cards are final sale and cannot be returned...

Question: Can I return a gift card?
```

## Worked example: sizing the pieces

A help centre has 400 articles, average 1,200 words each. With chunks of about 300 words and 50 words of overlap, each article yields about 5 chunks, so the index holds roughly 2,000 vectors. With a 384-dimension embedding model that is about 3 MB of 32-bit floats: a NumPy array or a small library (FAISS, Chroma, or the `pgvector` extension for PostgreSQL) is plenty. You only need a dedicated vector database at millions of chunks or with heavy concurrent traffic.

## Design decisions that matter

- **Chunk size:** small chunks retrieve precisely but lose context; large chunks carry context but dilute the match. Start around 200–500 tokens and test. Split on headings and paragraphs rather than fixed character counts.
- **Metadata:** keep the source, section title and date with each chunk. Use it to filter (only current policies, only this customer's region) and to cite.
- **Hybrid search:** embeddings capture meaning but can miss exact terms such as product codes or error numbers. Combining them with keyword search (BM25) usually helps.
- **Re-ranking:** retrieve 20–50 candidates cheaply, then re-order them with a cross-encoder model and keep the best 5.
- **Access control:** filter by the user's permissions **before** retrieval. A RAG system must never show someone a document they could not open themselves.
- **Freshness:** re-index when documents change, and remove deleted ones.

## Evaluate retrieval separately from generation

When an answer is wrong, find out which half failed:

- **Retrieval:** was the right chunk in the top-k? Measure **recall@k** on a set of questions where you know which source holds the answer.
- **Generation:** given the right chunks, did the model answer correctly, cite correctly, and refuse when the answer was missing (**faithfulness**)?

If recall@5 is 60%, no prompt engineering will fix the answers. Improve chunking, search or metadata first. The next lesson builds the evaluation set.

## Risks

- **Hallucination still happens.** Instruct the model to use only the sources and to say "I don't know", then test that it does.
- **Prompt injection:** a retrieved document can contain instructions ("ignore the above and..."). Treat retrieved text as data, keep system instructions separate, and limit what the app can do with the output.
- **Privacy:** sending internal documents to a hosted LLM shares them with the provider. Check policy, or use an open-weight model locally.

## Common mistakes

- **Different embedding models** for the index and the query.
- **No overlap**, so the key sentence is split across two chunks.
- **Judging the system on five hand-picked questions.**
- **Ignoring tables and PDFs**: bad text extraction is the most common silent failure. Read some chunks yourself.

## Quick recap

- RAG = embed and index your documents, retrieve the top-k chunks for each question, and generate an answer grounded in them, with citations.
- Chunking, metadata, hybrid search and access control matter more than model choice.
- Measure retrieval (recall@k) and generation (correctness, faithfulness) separately.

## Try it

Open the **Do** tab. You will design a RAG pipeline for a small document set of your choice and test its retrieval in a free notebook, with no paid API needed.


Design a RAG pipeline that retrieves relevant passages from your documents and grounds an LLM's answers in them.

Retrieval-augmented generation

What is Retrieval-Augmented Generation (RAG)?

RAG Explained

## Why this matters

"It looked good when I tried it" is not evaluation. A model or LLM app that seemed fine on five questions can fail on a whole category of real inputs, and a prompt tweak that fixes one case can quietly break ten others. A fixed **evaluation set** with an automatic score lets you compare models, prompts and settings honestly, catch regressions before users do, and say with evidence whether a change helped. Building one is often the highest-value work on an ML project.

## The core idea: a fixed, representative test set plus a metric

An evaluation set is a list of cases. Each case has an **input**, a **reference** (the expected answer, label, or the source that should be retrieved) and **tags** that describe it:

```python
{"id": "q4", "question": "Can I return a gift card?",
 "reference": "No, gift cards are final sale", "tags": ["returns"]}
```

You run the system on every input, score each output against its reference, and report the average overall **and per tag**. The per-tag view is where you learn things: "returns questions score 0.57, payments 0.95" tells you where to look.

## Where cases come from

- **Real traffic** (anonymised logs, past tickets): the best source, because it reflects what users actually ask.
- **Domain experts** writing questions and gold answers.
- **Known failures:** every bug report becomes a test case so it never comes back.
- **Edge cases on purpose:** typos and slang, very long inputs, questions the system should refuse or answer with "I don't know", other languages, adversarial inputs such as prompt injection.
- **Synthetic cases** generated by an LLM can add coverage, but review them by hand; they tend to be easier than real ones.

Aim for at least 50–100 cases to start, cover every category you care about, and keep the set **frozen** and versioned. If you tune prompts against it for weeks, you are overfitting to it, so keep a separate held-out set for the final check, just as with model selection.

## Metrics for different outputs

- **Classification** (labels): accuracy, precision, recall, F1 per class (Machine Learning Foundations).
- **Retrieval:** recall@k, the share of questions whose correct source is in the top k.
- **Short factual answers:** **exact match** after normalisation, and **token F1**, the overlap between predicted and reference words.
- **Open-ended text** (summaries, explanations): human rating with a clear rubric, or **LLM-as-a-judge**, where a strong model grades each answer against the reference and rubric. Judges are useful at scale but have biases (they favour longer answers, and their own style), so check a sample of their grades against human ones.
- **Operational:** latency, cost per request, refusal rate, and format validity (did the JSON parse?).

### Worked example: token F1

Normalise both strings (lowercase, strip punctuation and the articles "a", "an", "the"), split into tokens, and count the shared tokens:

```text
prediction: "the cat sat"      -> [cat, sat]
reference:  "a cat sat down"   -> [cat, sat, down]
shared = 2
precision = 2/2 = 1.0    recall = 2/3 = 0.67
F1 = 2 * 1.0 * 0.67 / (1.0 + 0.67) = 0.8
```

Exact match would score this 0. Token F1 gives partial credit, but it also has blind spots: "Usually 3-5 business days" against "3 to 5 business days" scores only 0.44 although both mean the same thing. That is why you **read the failures**: a metric tells you where to look, not what is true.

```python
from collections import Counter

def token_f1(pred_tokens, ref_tokens):
    common = sum((Counter(pred_tokens) & Counter(ref_tokens)).values())
    if common == 0:
        return 0.0
    p, r = common / len(pred_tokens), common / len(ref_tokens)
    return 2 * p * r / (p + r)
```

## Tools

Simple evaluation needs only a CSV or JSON file and a loop like the one in the activity. As projects grow, libraries help: scikit-learn metrics for classic models, Hugging Face `evaluate`, and open-source LLM evaluation tools such as Ragas (for RAG), promptfoo or DeepEval. Experiment trackers (next module) store each evaluation run so you can compare them. Run the evaluation in CI so every prompt or model change is scored automatically.

## Common mistakes

- **Too few or too easy cases**, or all from one category.
- **Changing the evaluation set** between the runs you compare.
- **One headline number only**: always break down by tag and read the worst cases.
- **Trusting an LLM judge blindly** without checking it against human grades.
- **Testing at a high temperature with one run**: results vary, so use low temperature or average several runs.
- **Personal data in the set**: anonymise real logs before they become test fixtures.

## Quick recap

- An evaluation set is a frozen, versioned list of inputs, references and tags.
- Pick metrics that fit the output: classification metrics, recall@k, exact match and token F1, rubric or judge scores, plus latency and cost.
- Report overall and per tag, and read the failures by hand.
- Every bug becomes a test case; keep a held-out set to avoid overfitting your prompts.

## Try it

Open the **Do** tab. You will write normalisation, exact match, token F1 and an evaluation report in plain Python, add edge cases to a help-desk evaluation set, and catch a wrong answer automatically.


Build a fixed evaluation set with references and tags, and score a model or LLM app with exact match and token F1.

Evaluating models and LLM apps

Introduction to LLM Evaluations – Model Evals vs Application Evals | CampusX

LLM as a Judge: Scaling AI Evaluation Strategies

Understand how large language models work, ground them in your documents with retrieval-augmented generation, and evaluate models and LLM apps with a fixed test set.