Go from classic ML to the models used at work: neural networks, embeddings, CNNs, gradient boosting and LLM apps, plus evaluation, experiment tracking and a capstone.

Applied Machine Learning

Create a project folder, start a git repository, and create a virtual environment. Install the tools: `pip install mlflow scikit-learn`.

Write `train.py` that loads a built-in dataset (for example `sklearn.datasets.load_breast_cancer`), makes a fixed train/validation split with `random_state=0`, trains a model and computes a validation metric.

Wrap the training in `mlflow.start_run()` and log the parameters (`mlflow.log_params`), the metric (`mlflow.log_metric`), a tag with the data source and split seed, and the trained model.

Commit the code, then log the git commit hash as a tag (for example from `git rev-parse HEAD`).

Run at least six experiments that change one setting at a time (for example three learning rates and two tree depths, or two model families).

Open the UI with `mlflow ui` and go to http://127.0.0.1:5000 (in Colab, use `mlflow.search_runs()` to get the runs as a pandas DataFrame instead). Compare the runs side by side.

Re-run your best configuration with three different model seeds, or with 5-fold cross-validation, and log the mean and standard deviation of the metric.

Write a short note in the run description or your README: which run you would choose, why, and whether the difference from the runner-up is larger than the seed-to-seed variation.

## Why this matters

A real ML project runs dozens or hundreds of experiments: different features, models, learning rates, seeds and data versions. Two weeks in, someone asks "which run produced the model we showed the client, and what data did it use?" Without records, nobody knows, and the result cannot be reproduced. **Experiment tracking** records every run automatically, so you can compare results, reproduce the best one and explain how you got there. Teams expect it of a mid-level data scientist or ML engineer.

## The core idea: every run is a record

A **run** is one execution of your training code. For each run, log:

- **Parameters:** the settings you chose (model type, learning rate, max depth, number of features).
- **Metrics:** the numbers that came out (validation AUC, RMSE, training time), optionally per epoch.
- **Artifacts:** files produced (the trained model, plots, the confusion matrix, a sample of predictions).
- **Context:** code version (git commit), data version (file hash or dataset tag), random seed, library versions and hardware.

Runs are grouped into **experiments**, for example "churn-model-v2".

## MLflow in a few lines

**MLflow** is a widely used open-source tracker. It runs locally for free, stores runs in a folder (or a database) and gives you a web UI. Install it in a virtual environment with `pip install mlflow scikit-learn`.

```python
import mlflow
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.25, random_state=0)

mlflow.set_experiment("breast-cancer-baseline")

for lr in [0.03, 0.1, 0.3]:
    with mlflow.start_run(run_name=f"hgb-lr-{lr}"):
        params = {"learning_rate": lr, "max_depth": 4, "random_state": 0}
        model = HistGradientBoostingClassifier(**params).fit(X_tr, y_tr)
        auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])
        mlflow.log_params(params)
        mlflow.log_metric("val_auc", auc)
        mlflow.set_tag("data", "sklearn breast_cancer, split seed 0")
        mlflow.sklearn.log_model(model, name="model")   # older MLflow: artifact_path="model"
```

Then run `mlflow ui` in the same folder and open http://127.0.0.1:5000. Select the runs and click **Compare** to see parameters and metrics side by side or plotted against each other. `mlflow.autolog()` can log most parameters and metrics for scikit-learn, XGBoost, LightGBM, PyTorch Lightning and Keras automatically.

Other trackers follow the same model: **Weights & Biases** (hosted, free tier for individuals), **Neptune**, **Comet**, **TensorBoard** for training curves, and **DVC** for versioning data alongside git. The concepts transfer.

## Worked example: reading a comparison

| run | learning_rate | max_depth | val_auc | train_time_s |
|---|---|---|---|---|
| hgb-lr-0.03 | 0.03 | 4 | 0.991 | 0.9 |
| hgb-lr-0.1 | 0.1 | 4 | 0.993 | 0.4 |
| hgb-lr-0.3 | 0.3 | 4 | 0.989 | 0.3 |

(Illustrative numbers.) The differences here are a few thousandths, which is well within the noise of a 143-row validation set. Before declaring a winner, repeat with several seeds or use cross-validation and log the mean and standard deviation. Tracking makes this cheap: it is just more runs.

## Good habits

- **Log from code, not by hand.** Copying numbers into a spreadsheet goes wrong.
- **Commit before you run** and log the commit hash, so code and result match. Never compare runs where code changed silently.
- **Version the data.** Log a hash of the training file or a dataset version tag. A model is only reproducible if its data is.
- **Fix and log seeds** for the split and the model.
- **Name runs and add a short note** on what you were testing ("added tenure buckets").
- **Log failures too.** A record of what did not work stops teammates repeating it.
- **Promote deliberately.** MLflow's **model registry** lets you mark a version as the candidate for production, with an audit trail (the MLOps course covers this).
- **Keep secrets out.** Do not log API keys, and avoid logging raw personal data as artifacts.

## Common mistakes

- Choosing the best run by test-set score (use validation; test once at the end).
- Comparing runs trained on different data splits.
- Tracking metrics but not parameters, so the best result cannot be rebuilt.
- Leaving a hosted tracker's project public when it contains company data.

## Quick recap

- A run records parameters, metrics, artifacts and context (code commit, data version, seed, environment).
- MLflow runs locally for free: `start_run`, `log_params`, `log_metric`, `log_model`, then `mlflow ui` to compare.
- Small metric differences need repeated runs or cross-validation before they mean anything.
- Version code and data so the best run can be reproduced.

## Try it

Open the **Do** tab. You will install MLflow locally (or in a free notebook), log at least six runs of a model, compare them in the UI and write down which one you would choose and why.


Log parameters, metrics, artifacts and context for every training run with MLflow, and compare runs to pick a model.

Experiment tracking

Never lose a model again with MLflow Experiment Tracking

MLFlow Tutorial | ML Ops Tutorial

Fill in the model details: name, version, date, owner, model type and library versions, and the code commit or notebook that trained it.

Write the intended use (the decision it supports and who uses it) and at least two explicit out-of-scope uses.

Describe the training and evaluation data: source, time period, size, how labels were produced, and known gaps.

Report the main metric with an uncertainty estimate (for example a bootstrap interval) next to a baseline such as predict-the-majority-class.

Add a performance breakdown for at least one relevant subgroup or segment, and note any gap and what you would do about it.

Translate the metric into consequences at your chosen threshold: counts of correct flags, misses and false alarms out of a realistic number of cases.

List the top drivers of predictions in plain words (from permutation importance or SHAP) without causal claims.

Write limitations, risks and a maintenance plan: what happens when the model is wrong, how it will be monitored, and when it will be retrained.

Write a five-sentence plain-English summary for a non-technical manager at the top of the card, and ask someone outside data science to read it and tell you what they think the model does.

## Why this matters

A model nobody understands does not get used, and a model people misunderstand gets used for the wrong things. Data scientists and ML engineers spend a lot of their time explaining: to a product manager deciding whether to launch, to a compliance reviewer asking how errors affect customers, to the engineer who will maintain the model next year. The standard written artefact is the **model card**: a short document that says what the model does, how well, for whom, and where it should not be used.

## The core idea: a model card

Model cards were proposed by Mitchell et al. in "Model Cards for Model Reporting" (2019) and are now common practice; Hugging Face shows one on every model page. A good card is one or two pages and covers:

1. **Model details:** name, version, date, owner and contact, model type (for example "gradient-boosted trees, LightGBM 4"), training code commit.
2. **Intended use:** the decision it supports, the intended users, and **out-of-scope uses** ("not for credit decisions", "not validated for patients under 18").
3. **Data:** where training and evaluation data came from, time period, size, known gaps, and how labels were produced.
4. **Metrics:** the main metric with its uncertainty, a baseline for comparison, and the decision threshold if there is one.
5. **Breakdown by group:** performance for relevant subgroups (region, device, age band, language), as practised in the fairness lesson of Machine Learning Foundations.
6. **Limitations and risks:** known failure modes, what happens when the model is wrong, and the mitigations (human review, fallbacks).
7. **Maintenance:** how it is monitored, when it will be retrained, and who to tell about problems.

## Translate metrics into consequences

Stakeholders do not act on "AUC 0.87". They act on what the model will do to their numbers. Turn metrics into counts and costs at the chosen threshold.

### Worked example

A churn model flags customers for a retention call. On 10,000 recent customers, 800 actually churned. At the chosen threshold the model flags 1,000 customers, 480 of whom churned.

```text
precision = 480 / 1,000 = 48%   (about half of the calls reach someone at risk)
recall    = 480 / 800   = 60%   (we reach 6 in 10 of the customers who would leave)
random calling of 1,000 customers would reach about 80 churners (8%)
```

The plain-English version: "Each month the team can call 1,000 customers. Calling the ones this model flags reaches about 480 customers who would otherwise leave, six times as many as calling at random. It still misses about 4 in 10 churners, mostly customers in their first month, for whom we have little data."

That paragraph states the decision, the gain over a baseline, the main limitation and who is affected. It is worth more than a page of metrics.

## Explaining what drives predictions

- **Global explanations:** which features matter most overall. Use permutation importance or mean absolute SHAP values, and present the top five or so in plain words ("recent support tickets", not `n_tix_30d`).
- **Local explanations:** why this one prediction. SHAP values show which features pushed one customer's score up or down. Useful for reviewers and for debugging.
- **Be careful with causal language.** "Customers with more support tickets are flagged more often" is accurate. "Support tickets cause churn" is a claim the model cannot support.

## Communicating uncertainty

- Give ranges, not just points: "AUC 0.87 (95% interval 0.85 to 0.89, from bootstrap resampling)".
- Say how the evaluation was done: time period, holdout or cross-validation, and whether the test data came after the training data.
- Say what would change the conclusion: a new product launch, a pricing change, a different customer mix.

## Formats that work

- **One-page summary first:** decision, recommendation, key number, main risk. Details go in an appendix or the full card.
- **Charts with one message each:** precision and recall against the threshold, performance by group, calibration (do predicted 30% risks churn about 30% of the time?).
- **Versioned in the repository** next to the code (`MODEL_CARD.md`), or in the model registry, so it is updated with each release.

## Common mistakes

- Reporting only the best-looking metric, or a metric on training data.
- Leaving out a baseline, so nobody can tell if the model is any good.
- Omitting out-of-scope uses, so the model is reused for a decision it was never validated for.
- Hiding subgroup gaps instead of stating them with a plan.
- Jargon without translation ("F1 of 0.53") for a non-technical audience.
- Writing the card once and never updating it.

## Quick recap

- A model card states the model's purpose, data, metrics, subgroup performance, limitations and maintenance plan, including what it must not be used for.
- Translate metrics into counts, costs and comparisons with a baseline.
- Explain drivers with permutation importance or SHAP, without causal claims.
- Report uncertainty, and keep the card versioned with the model.

## Try it

Open the **Do** tab. You will write a one-page model card for a model you built in this course (or a public one), including a plain-English summary for a non-technical reader.


Write a model card and explain a model's results, limits and risks in terms a decision-maker can act on.

Communicating model results

What are @HuggingFace Model Cards?

What is Explainable AI?

Write a one-paragraph problem statement: the decision, the users, what the model predicts, the metric that matches the decision, and what "good enough" means.

Choose and download a public dataset (Kaggle, UCI Machine Learning Repository, Hugging Face Datasets or a government open-data portal), and record its source and licence in `data/README.md`.

Explore the data: distributions, missing values, label balance and possible leakage columns. Write down what you found.

Choose a split that matches how the model will be used (random, grouped or time-based), and lock the test set away.

Build two baselines (predict-the-majority or the mean, plus a simple linear or TF-IDF model) and record their validation scores.

Improve with at least one technique from this course (gradient boosting, embeddings, transfer learning or RAG), tracking every run with MLflow and changing one thing at a time.

Evaluate the final model once on the test set with a bootstrap interval and a breakdown by at least one subgroup.

Read 20 to 50 errors, group them into named categories, and write what each category suggests you try next.

Write the README (problem, data, approach, results table against baselines, error analysis, limitations, how to reproduce) and a MODEL_CARD.md, with pinned `requirements.txt` and a script that retrains the final model.

Ask a peer to clone the repository and reproduce your result from the README, fix whatever blocked them, and share the finished project in the course forum.

## Why this matters

Hiring managers for data science and ML engineering roles want evidence that you can take a vague problem to a working, honest result. A capstone project is that evidence. One complete, well-documented project beats five tutorial notebooks: it shows you can frame a problem, handle messy data, choose a sensible baseline, evaluate without fooling yourself, and explain what you found. This lesson gives you a plan; the **Do** tab walks you through it.

## The core idea: a small problem, done end to end

Scope is the most common reason capstones fail. Pick a problem you can finish in roughly 20 to 40 hours. The end-to-end chain matters more than a fancy model:

```text
question -> data -> baseline -> model -> evaluation -> error analysis -> write-up -> (optional) demo
```

### Choosing a problem

Good capstones have:

- **A decision attached.** "Predict which support tickets need a specialist, so they skip the first queue" is better than "classify tickets".
- **Public, legally usable data.** Sources: Kaggle datasets, the UCI Machine Learning Repository, Hugging Face Datasets, government open-data portals (data.gov, data.gov.uk, Eurostat), or data you collect yourself. Read the licence, and avoid personal data you have no right to use.
- **Enough rows** for an honest test set: aim for at least a few thousand for tabular problems.
- **A domain you care about**, so you can judge whether results make sense.

Example scopes that fit this course:

| idea | type | techniques from this course |
|---|---|---|
| Predict bike-share demand per hour from weather and calendar | tabular regression | gradient boosting, time-based split |
| Route customer complaints to the right department | text classification | embeddings + logistic regression, TF-IDF baseline |
| Detect defective parts from product photos | image classification | transfer learning, augmentation |
| Answer questions over a public handbook | LLM + retrieval | RAG, evaluation set |

## The project plan

1. **Frame it.** Write a one-paragraph problem statement: who decides what, what the model predicts, and the metric that matches the decision. Define "good enough" up front.
2. **Get and understand the data.** Explore distributions, missing values and label balance. Write down how labels were produced and any obvious biases.
3. **Split before you model.** Random, grouped (by customer, patient or product) or time-based, depending on how the model will be used. Lock the test set away.
4. **Baselines first.** Predict the mean or the majority class, then a simple model (logistic or linear regression, TF-IDF). Every later result is compared with these.
5. **Improve deliberately.** One change per run, tracked with MLflow (module 4, lesson 1). Use cross-validation or a validation set for every choice.
6. **Evaluate once on the test set**, with an uncertainty estimate (bootstrap interval) and a breakdown by relevant groups.
7. **Error analysis.** Read 20–50 errors. Group them ("short messages", "night-time hours", "blurry photos"). This is often the most insightful part of the write-up.
8. **Write it up.** A README with the problem, data, approach, results table, error analysis, limitations and how to reproduce the result, plus a model card (lesson 2).
9. **Optional demo.** A small Streamlit or Gradio app, or a Hugging Face Space (free tier), lets a reviewer try the model in a minute.

## What a reviewer looks for

- A clear **problem statement and metric**, tied to a decision.
- **No leakage**, an honest split, and the test set used once.
- **Baselines** and a results table that compares against them.
- **Reproducibility:** pinned dependencies (`requirements.txt`), fixed seeds, a script or notebook that runs top to bottom, data download instructions.
- **Readable code:** functions instead of one giant cell, sensible names, no secrets committed.
- **Judgement:** limitations stated plainly, and next steps that follow from the error analysis.

## Repository layout that works

```text
capstone/
  README.md            problem, results, how to run
  MODEL_CARD.md
  requirements.txt     pinned versions
  data/README.md       where to download the data (do not commit large or licensed files)
  notebooks/01_eda.ipynb
  src/train.py         python src/train.py reproduces the final model
  src/evaluate.py
  reports/figures/
```

## Compute and cost

Everything in this plan runs on a laptop or in free Colab and Kaggle notebooks. If you choose a hosted LLM API or cloud resources, check the pricing first, set a spending limit or budget alert, prefer free tiers, and delete resources when you finish. Never commit API keys; use environment variables.

## Common mistakes

- **Scope creep:** starting a second model family before the first one is evaluated.
- **Leakage** through random splits of time series, or through features computed on the full dataset.
- **Only accuracy on imbalanced data.**
- **No baseline**, so a 92% score means nothing.
- **A notebook that only runs on your machine** because of hidden state or absolute paths.
- **Skipping the write-up**, which is what reviewers actually read.

## Quick recap

- Pick a small problem with a real decision, usable data and a clear metric.
- Split first, build baselines, improve one change at a time and track every run.
- Test once, analyse errors, and write a README plus model card that a reviewer can follow and reproduce.

## Try it

Open the **Do** tab and work through the capstone checklist. Share the finished repository in the course forum for feedback.


Plan, build, evaluate and document an end-to-end machine learning project for your portfolio.

Capstone project

Improving Model Performance (C3W1L01)

How to Build a Machine Learning Portfolio That Gets You Hired!

Track experiments, communicate model results with a model card, and deliver an end-to-end capstone project.