Experiment tracking

Why this matters

A real ML project runs dozens or hundreds of experiments: different features, models, learning rates, seeds and data versions. Two weeks in, someone asks "which run produced the model we showed the client, and what data did it use?" Without records, nobody knows, and the result cannot be reproduced. Experiment tracking records every run automatically, so you can compare results, reproduce the best one and explain how you got there. Teams expect it of a mid-level data scientist or ML engineer.

The core idea: every run is a record

A run is one execution of your training code. For each run, log:

• Parameters: the settings you chose (model type, learning rate, max depth, number of features).
• Metrics: the numbers that came out (validation AUC, RMSE, training time), optionally per epoch.
• Artifacts: files produced (the trained model, plots, the confusion matrix, a sample of predictions).
• Context: code version (git commit), data version (file hash or dataset tag), random seed, library versions and hardware.

Runs are grouped into experiments, for example "churn-model-v2".

MLflow in a few lines

MLflow is a widely used open-source tracker. It runs locally for free, stores runs in a folder (or a database) and gives you a web UI. Install it in a virtual environment with pip install mlflow scikit-learn.

import mlflow
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.25, random_state=0)

mlflow.set_experiment("breast-cancer-baseline")

for lr in [0.03, 0.1, 0.3]:
    with mlflow.start_run(run_name=f"hgb-lr-{lr}"):
        params = {"learning_rate": lr, "max_depth": 4, "random_state": 0}
        model = HistGradientBoostingClassifier(**params).fit(X_tr, y_tr)
        auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])
        mlflow.log_params(params)
        mlflow.log_metric("val_auc", auc)
        mlflow.set_tag("data", "sklearn breast_cancer, split seed 0")
        mlflow.sklearn.log_model(model, name="model")   # older MLflow: artifact_path="model"

Then run mlflow ui in the same folder and open http://127.0.0.1:5000. Select the runs and click Compare to see parameters and metrics side by side or plotted against each other. mlflow.autolog() can log most parameters and metrics for scikit-learn, XGBoost, LightGBM, PyTorch Lightning and Keras automatically.

Other trackers follow the same model: Weights & Biases (hosted, free tier for individuals), Neptune, Comet, TensorBoard for training curves, and DVC for versioning data alongside git. The concepts transfer.

Worked example: reading a comparison

(Illustrative numbers.) The differences here are a few thousandths, which is well within the noise of a 143-row validation set. Before declaring a winner, repeat with several seeds or use cross-validation and log the mean and standard deviation. Tracking makes this cheap: it is just more runs.

Good habits

• Log from code, not by hand. Copying numbers into a spreadsheet goes wrong.
• Commit before you run and log the commit hash, so code and result match. Never compare runs where code changed silently.
• Version the data. Log a hash of the training file or a dataset version tag. A model is only reproducible if its data is.
• Fix and log seeds for the split and the model.
• Name runs and add a short note on what you were testing ("added tenure buckets").
• Log failures too. A record of what did not work stops teammates repeating it.
• Promote deliberately. MLflow's model registry lets you mark a version as the candidate for production, with an audit trail (the MLOps course covers this).
• Keep secrets out. Do not log API keys, and avoid logging raw personal data as artifacts.

Common mistakes

• Choosing the best run by test-set score (use validation; test once at the end).
• Comparing runs trained on different data splits.
• Tracking metrics but not parameters, so the best result cannot be rebuilt.
• Leaving a hosted tracker's project public when it contains company data.

Quick recap

• A run records parameters, metrics, artifacts and context (code commit, data version, seed, environment).
• MLflow runs locally for free: start_run, log_params, log_metric, log_model, then mlflow ui to compare.
• Small metric differences need repeated runs or cross-validation before they mean anything.
• Version code and data so the best run can be reproduced.

Try it

Open the Do tab. You will install MLflow locally (or in a free notebook), log at least six runs of a model, compare them in the UI and write down which one you would choose and why.