Learn how machines learn from data: implement regression, trees, metrics, k-means, PCA and anomaly detection from scratch in NumPy, and practise honest evaluation.

Machine Learning Foundations

## Why this matters

Models only see numbers. The work of turning raw columns (text categories, dates, prices in different units, free-text notes) into useful numeric inputs is called **feature engineering**, and on tabular data it often matters more than the choice of algorithm. A k-nearest-neighbours model can go from coin-flip to strong just by encoding and scaling the same information properly.

## Encoding categories

A column like `neighbourhood = north | center | south` must become numbers.

**Ordinal (integer) encoding** maps each category to an integer: center=0, north=1, south=2. It is compact, but it invents an order and distances that do not exist: the model now thinks "south" is twice as far from "center" as "north" is. Use it only when the categories really are ordered (`small < medium < large`, education levels), or with tree models, which can cope with arbitrary codes.

**One-hot encoding** creates one 0/1 column per category:

```text
neighbourhood   is_center  is_north  is_south
center              1          0         0
south               0          0         1
north               0          1         0
```

Every category is now equally far from every other, which is what distance-based and linear models need.

```python
import numpy as np

def one_hot(values, categories):
    values = np.asarray(values)
    return np.column_stack([(values == c).astype(float) for c in categories])
```

Practical points:

- **Fix the category list on the training data.** A category seen only at prediction time should map to all zeros (or an explicit "other" column), not crash the pipeline.
- **High-cardinality columns** (ZIP codes, product IDs with thousands of values) explode into thousands of columns. Group rare values into "other", or use **target encoding** (replace each category by the average label in training data, computed inside cross-validation folds to avoid leakage).
- For linear models, drop one column per feature (`drop="first"`) if you need unique coefficients; regularised models usually do not need this.

## Scaling numeric features

Distance-based models (k-NN, k-means, SVMs), PCA and anything trained by gradient descent with regularisation are sensitive to feature scale. A price in dollars (around 1,500 ± 400) will completely swamp a distance in kilometres (0–10) in a Euclidean distance.

- **Standardisation**: `(x - mean) / sd`, giving mean 0 and sd 1. The usual default.
- **Min-max scaling**: `(x - min) / (max - min)`, giving 0 to 1. Sensitive to outliers.
- **Log transform**: for skewed positive values such as income, counts or prices, `log(1 + x)` often helps both linear and distance-based models.

As in Lesson 2, compute the means, standard deviations and category lists **on the training data only**. Tree models do not need scaling.

## k-nearest neighbours: a scale-sensitive model

k-NN is the simplest classifier: to predict a new row, find the `k` closest training rows and take a majority vote of their labels.

```python
d = ((X_query[:, None, :] - X_train[None, :, :]) ** 2).sum(axis=2)  # all distances
nearest = np.argsort(d, axis=1)[:, :k]
pred = (y_train[nearest].mean(axis=1) >= 0.5).astype(int)
```

There is no training step, so everything depends on what "close" means, which is exactly what feature engineering controls.

### Worked example

Listings are booked mostly when they are close to the centre, and the centre neighbourhood is especially popular; price is irrelevant in this dataset. With raw features `[distance_km, price, neighbourhood_code]`, the nearest neighbours are simply the listings with similar **prices**, so accuracy sits near the 50% you would get by guessing. After standardising distance and price and one-hot encoding the neighbourhood, distance and location drive the neighbours, and accuracy rises to around 90%. Same data, same model, different features.

## Other common features

- **Dates**: extract day of week, month, hour, "is holiday", and time since an event (days since signup). Never use the raw timestamp as a number.
- **Ratios and interactions**: debt / income, price per square metre, clicks / impressions.
- **Aggregates**: a customer's order count in the last 30 days (computed only from data before the prediction time).
- **Missing values**: impute (median for numbers, "missing" category for categories) and add an "was missing" indicator, since missingness itself can be informative.

## In scikit-learn

```python
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline

pre = ColumnTransformer([
    ("num", StandardScaler(), ["distance_km", "price"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["neighbourhood"]),
])
model = Pipeline([("pre", pre), ("knn", KNeighborsClassifier(n_neighbors=7))])
model.fit(X_train_df, y_train)          # every step is fitted on training data only
```

Putting preprocessing inside a `Pipeline` is the reliable way to avoid leakage, especially during cross-validation (Lesson 12).

## Common mistakes

- **Integer-encoding unordered categories** for linear or distance-based models.
- **Scaling with statistics from the full dataset.**
- **Features built with information from after the prediction time.**
- **Crashing on unseen categories in production.** Use `handle_unknown="ignore"` or an "other" bucket.
- **Engineering hundreds of features without validation**, which invites overfitting.

## Quick recap

- One-hot encode unordered categories; ordinal-encode only real orders.
- Scale numeric features for distance-, gradient- and regularisation-based models; trees do not need it.
- Fit encoders and scalers on training data, inside a pipeline.
- Good features often beat a fancier model.

## Try it

On the Do tab, write one-hot encoding, a training-only standardiser and k-NN, and compare accuracy with raw versus engineered features.


Encode categories, scale numeric features and build useful inputs, then see the effect on a k-NN classifier.

Feature engineering

One-Hot, Label, Target and K-Fold Target Encoding, Clearly Explained!!!

## Why this matters

Models are increasingly used to decide who gets a loan, a job interview, an apartment, extra medical attention or a police visit. A model learns from historical data, and history contains unequal treatment, missing groups and flawed measurements. Without checking, a model can quietly repeat or amplify those problems at scale. Checking a model for group disparities is part of doing the job properly, and in many domains (credit, hiring, housing) it is also a legal obligation.

## Where bias comes from

- **Historical bias**: the labels reflect past decisions. If past hiring favoured one group, a model trained to predict "was hired" learns that preference.
- **Representation bias**: some groups are under-represented in the training data, so the model is less accurate for them. Face-analysis systems trained mostly on lighter-skinned faces made far more errors on darker-skinned faces.
- **Measurement bias**: the label is a poor proxy for what you care about. Using past health-care **spending** as a proxy for health **need** underestimates need for groups that historically had less access to care.
- **Aggregation bias**: one model for everyone when relationships differ between groups.
- **Deployment bias**: a model used for a purpose or population it was not built for.

## Removing the sensitive column is not enough

Dropping race or gender from the features ("fairness through unawareness") rarely works. Other features act as **proxies**: ZIP code, school, first name, shopping patterns and gaps in employment history can all correlate strongly with protected attributes. You usually need the sensitive attribute **for auditing**, even if the model never uses it.

## Measuring disparities

Compute the usual metrics **separately for each group** and compare. The most common group fairness criteria:

- **Demographic parity**: each group gets positive decisions at a similar rate (similar **selection rates**).
- **Equal opportunity**: among people who truly qualify, each group is approved at a similar rate (similar **true positive rates**, that is, recall).
- **Equalised odds**: similar true positive rates **and** similar false positive rates.
- **Calibration**: among people given a 70% score, about 70% truly have the outcome, in every group.

### Worked example: a loan model

```text
                      group A    group B
applicants              1,000        500
approved                  400        120
selection rate           0.40       0.24
truly creditworthy        600        250
approved & creditworthy   380        110
true positive rate       0.63       0.44
```

The **disparate impact ratio** is `0.24 / 0.40 = 0.60`. In US employment law, the EEOC's "four-fifths rule" treats a ratio below 0.8 as a sign of possible adverse impact that needs investigating. The equal opportunity gap is `0.63 - 0.44 = 0.19`: a creditworthy applicant in group B is much less likely to be approved than one in group A. Both numbers say this model needs a closer look before anyone relies on it.

## You cannot satisfy every definition at once

When groups have different base rates, it is mathematically impossible (except in trivial cases) to have calibration, equal false positive rates and equal false negative rates all at the same time. Choosing which criterion matters is a **value judgement** about which errors harm people most, and it belongs with domain experts, affected communities, legal and compliance staff, not with the data scientist alone. Your job is to measure clearly, explain the trade-offs and document the decision.

## What you can do

1. **Audit**: report accuracy, selection rate, TPR and FPR per group (and for intersections such as age × gender) on held-out data.
2. **Fix the data**: collect more data for under-represented groups, fix label definitions, remove proxy features that have no legitimate reason to be there.
3. **Adjust the model or thresholds**: fairness-constrained training, or group-aware thresholds where the law allows it (in some domains it does not; check).
4. **Keep a human in the loop** for high-stakes decisions and provide a way to appeal.
5. **Document**: a **model card** states the intended use, the data, performance per group and known limitations.
6. **Monitor** after deployment, because populations and behaviour drift.

Open-source tools help with the arithmetic: **Fairlearn** (`MetricFrame` computes any metric by group) and IBM's **AI Fairness 360**. Frameworks such as the NIST AI Risk Management Framework (https://www.nist.gov/itl/ai-risk-management-framework) describe how organisations govern these risks. Rules differ by country and sector and keep changing (for example the EU AI Act), so confirm current requirements with your legal team. Checked on 2026-10-23.

## Common mistakes

- **Reporting one overall accuracy number.** A model can be 95% accurate overall and much worse for a small group.
- **Assuming dropping the sensitive column makes the model fair.**
- **Treating fairness as a one-off check** instead of something monitored.
- **Choosing a fairness metric to make the numbers look good** rather than the one that matches the harm.
- **Ignoring small groups** because their sample sizes are noisy; report uncertainty instead of leaving them out.

## Quick recap

- Bias enters through history, representation, measurement, aggregation and deployment.
- Proxies mean that removing a sensitive column does not remove its influence.
- Compare selection rates, TPR and FPR across groups; the four-fifths ratio and TPR gaps are simple flags.
- Fairness definitions conflict; choose with stakeholders, document with a model card, keep monitoring.

## Try it

Take the quiz on the Do tab to practise computing and interpreting group disparities.


Find where bias enters a model and check it for group disparities with selection rates and error rates.

Fairness and bias

Machine Learning Crash Course: Fairness

3 types of bias in AI | Machine learning

## Why this matters

Most models have **hyperparameters**: settings you choose before training rather than parameters the model learns. The regularisation strength of a ridge regression, the depth of a tree, `k` in k-NN, the number of trees in a forest, the learning rate of gradient descent. Picking them well can matter as much as picking the model. Picking them **honestly**, without peeking at the test set, is what keeps your reported performance true.

## The problem with one validation split

In Lesson 2 you held out one validation set. With a small dataset, that single split is noisy: a different random split can rank the settings differently, and you may tune to the quirks of those particular rows. **k-fold cross-validation** uses the training data more efficiently and gives a more stable estimate.

## k-fold cross-validation

```text
shuffle the training rows and cut them into k folds (often k = 5 or 10)
for each fold i:
    train on the other k-1 folds
    score on fold i
CV score = average of the k scores
```

Every training row is used for validation exactly once and for training `k - 1` times. The spread of the `k` scores also tells you how uncertain the estimate is.

```python
import numpy as np

idx = np.random.default_rng(0).permutation(n)
folds = np.array_split(idx, 5)           # 5 nearly equal folds
for i in range(5):
    val_idx = folds[i]
    train_idx = np.concatenate(folds[:i] + folds[i + 1:])
```

The same rules as before apply: stratify folds for imbalanced classes, use group folds when rows belong to the same person, and use **forward-chaining** (expanding-window) splits for time series. Any preprocessing must be fitted **inside** each fold, which is why scikit-learn pipelines matter.

## Grid search

To tune, try a list of candidate values, compute the CV score of each, and pick the best. These are the numbers from this lesson's activity (ridge regression on 40 noisy points):

```text
alpha      CV MSE
1e-08      16.57   <- too flexible: fits the noise, wild between points
1e-04       0.14
1e-02       0.06   <- best
1           0.11
100         0.37   <- too rigid: underfits
```

Then **refit on the whole training set** with the chosen value and evaluate **once** on the test set.

Search on a logarithmic scale for strengths and rates (`1e-4, 1e-3, ... 10`) because their effects are multiplicative. For many hyperparameters at once, **random search** often finds good settings faster than a full grid, and Bayesian optimisation tools (such as Optuna) go further.

## Ridge regression and the bias-variance trade-off

The activity tunes **ridge regression**, which is linear regression plus a penalty `alpha * sum(w^2)` on large coefficients. The exact solution is

```text
w = (Xc^T Xc + alpha * I)^(-1) Xc^T yc        (Xc, yc centred, so the intercept is not penalised)
```

With 12 polynomial features of `x`, a tiny `alpha` lets the curve wiggle through every noisy point: low bias, high **variance**, and poor predictions on new data (overfitting). A huge `alpha` shrinks the curve toward a flat line: low variance, high **bias** (underfitting). Cross-validation finds the sweet spot between them. Every hyperparameter that controls flexibility (tree depth, `k` in k-NN, number of boosting rounds) is a version of this dial.

## Nested cross-validation

If you report the best CV score from a big search as your final performance, it is optimistic: you picked the maximum of many noisy numbers. Either evaluate once on an untouched test set (as here), or, when data is scarce, use **nested cross-validation**: an inner loop chooses hyperparameters, and an outer loop scores the whole tuning procedure.

## In scikit-learn

```python
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV, KFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

pipe = make_pipeline(PolynomialFeatures(12, include_bias=False), StandardScaler(), Ridge())
search = GridSearchCV(
    pipe, {"ridge__alpha": [1e-8, 1e-6, 1e-4, 1e-2, 1, 100]},
    cv=KFold(5, shuffle=True, random_state=0), scoring="neg_mean_squared_error")
search.fit(X_train, y_train)
print(search.best_params_, -search.best_score_)
print(search.score(X_test, y_test))           # once, at the end
```

`GridSearchCV` refits the best pipeline on all the training data automatically. `RandomizedSearchCV` does random search, and `cross_val_score` gives the k fold scores for one fixed setting.

## Common mistakes

- **Tuning on the test set**, directly or by trying "just one more" setting after seeing test results.
- **Preprocessing outside the folds** (for example scaling all training rows before cross-validation), which leaks validation statistics.
- **Shuffled folds on time series.**
- **Searching a linear grid** (0.1, 0.2, 0.3...) for a parameter that varies over orders of magnitude.
- **Ignoring the spread** of fold scores: if two settings differ by less than the fold-to-fold noise, prefer the simpler model.

## Quick recap

- Hyperparameters are chosen, not learned; choose them with cross-validation on training data.
- k-fold CV trains k times, validating on each fold once, and averages the scores.
- Grid or random search over log-scaled candidates, refit on all training data, test once.
- Regularisation strength trades variance against bias; CV finds the balance.

## Try it

On the Do tab, write k-fold splitting and cross-validated MSE, then let a grid search choose the ridge penalty.


Choose hyperparameters honestly with k-fold cross-validation and grid search, then test once.

Cross-validation and tuning

Machine Learning Fundamentals: Cross Validation

Machine Learning Tutorial Python - 16: Hyper parameter Tuning (GridSearchCV)

Turn raw columns into model-ready features, check models for unfair group disparities, and tune hyperparameters honestly with cross-validation.