Learn how machines learn from data: implement regression, trees, metrics, k-means, PCA and anomaly detection from scratch in NumPy, and practise honest evaluation.

Machine Learning Foundations

## Why this matters

Linear regression and logistic regression are the workhorses of applied machine learning. They train in milliseconds, are hard to overfit badly, and, most importantly, you can **explain** them: each feature gets one coefficient that says how it moves the prediction. Many production systems in banking, healthcare and marketing still use them for exactly that reason, and they are the baseline every fancier model has to beat.

## Linear regression

For a numeric label, linear regression predicts a weighted sum of the features:

```text
y_hat = b + w1*x1 + w2*x2 + ... + wp*xp        (in matrix form: y_hat = X @ w + b)
```

Training chooses `w` and `b` to minimise the mean squared error, exactly as in the gradient descent lesson. For linear regression there is also an exact solution (least squares), which NumPy computes directly:

```python
import numpy as np

A = np.column_stack([np.ones(len(X)), X])     # a column of 1s for the intercept
coef, *_ = np.linalg.lstsq(A, y, rcond=None)
b, w = coef[0], coef[1:]
```

### Interpreting coefficients

Suppose a model of apartment rent (in dollars per month) gives

```text
rent_hat = 400 + 2.1 * size_sqft + 150 * bedrooms - 12 * minutes_to_station
```

Read each coefficient as: **holding the other features fixed**, one more unit of this feature changes the prediction by this amount. One extra minute from the station is associated with $12 less rent, for apartments of the same size and bedroom count.

Three cautions:

- Coefficients depend on **units**. Measure size in square metres and its coefficient grows about 10.8 times. To compare importance across features, standardise them first.
- **Correlated features** share credit unpredictably. If size and bedrooms move together, their individual coefficients become unstable even when predictions are fine.
- Coefficients describe **association**, not cause.

The usual quality measures are RMSE (typical error in the label's units) and R-squared (the share of variance explained).

## Logistic regression

For a yes/no label, we want a probability between 0 and 1. Logistic regression takes the same weighted sum and squashes it with the **sigmoid** function:

```text
z = X @ w + b                 (the log-odds, any real number)
p = sigmoid(z) = 1 / (1 + e^(-z))   (a probability)
```

`sigmoid(0) = 0.5`, large positive `z` gives probabilities near 1, large negative `z` near 0.

### The loss: log loss

Squared error works poorly for probabilities. Logistic regression minimises **log loss** (binary cross-entropy):

```text
log_loss = -mean( y*log(p) + (1-y)*log(1-p) )
```

It punishes confident wrong answers heavily: predicting `p = 0.01` for a true positive costs `-log(0.01) = 4.6`, while `p = 0.6` costs only 0.51. There is no closed-form solution, so we use gradient descent. Conveniently, the gradient has the same shape as for linear regression:

```text
err = p - y
dw = X.T @ err / n        db = mean(err)
```

### Interpreting logistic coefficients

A coefficient is the change in **log-odds** per unit of the feature. Exponentiate it to get an **odds ratio**. In a churn model, a coefficient of 0.7 on "support tickets last month" means each extra ticket multiplies the odds of churning by `e^0.7 ≈ 2.0`, holding other features fixed.

### From probability to decision

The model outputs probabilities; a **threshold** turns them into decisions. 0.5 is only the default. If missing a churner is expensive and a retention email is cheap, use a lower threshold such as 0.3. The next lesson shows how to choose it with metrics.

## Worked example in scikit-learn

```python
from sklearn.linear_model import LinearRegression, LogisticRegression

lin = LinearRegression().fit(X_train, y_rent)
print(lin.intercept_, lin.coef_)

clf = LogisticRegression().fit(X_train, y_churn)
proba = clf.predict_proba(X_test)[:, 1]      # P(class 1)
pred = (proba >= 0.3).astype(int)            # your own threshold
```

Note that scikit-learn's `LogisticRegression` applies **L2 regularisation** by default (`C=1.0`), which shrinks coefficients toward zero. That is usually good, but it means coefficients differ slightly from an unregularised fit, and it makes feature scaling matter.

## Common mistakes

- **Using linear regression for a 0/1 label.** It can predict probabilities below 0 or above 1. Use logistic regression.
- **Comparing raw coefficients across features with different units.**
- **Forgetting the intercept**, which forces the line through the origin.
- **Overflow in the sigmoid.** `np.exp(-z)` overflows for very negative `z`; clip `z` (for example to ±500) or use a library implementation.
- **Reading coefficients as causes.** They tell you what the model uses, not what would happen if you intervened.

## Quick recap

- Linear regression: weighted sum, minimise MSE, exact least-squares solution, coefficients in label units per feature unit.
- Logistic regression: weighted sum through a sigmoid, minimise log loss with gradient descent, coefficients in log-odds (exponentiate for odds ratios).
- The threshold is a business decision, not a fixed 0.5.

## Try it

On the Do tab, write the sigmoid and the log-loss gradient, train a logistic regression from scratch, and interpret its coefficients.


Fit linear regression for numbers and logistic regression for yes/no labels, and read their coefficients correctly.

Linear and logistic regression

Linear Regression, Clearly Explained!!!

StatQuest: Logistic Regression

## Why this matters

Linear models draw straight boundaries. Real data often has thresholds and interactions: a loan is risky if income is low **and** debt is high; a machine fails above a certain temperature. **Decision trees** capture these patterns naturally, and **random forests**, which average many trees, are one of the most reliable models for tabular data. They need little preprocessing, handle mixed feature types, and are a strong default when you need a good model fast.

## How a decision tree predicts

A tree is a flowchart of yes/no questions about features:

```text
income <= 42,000 ?
├── yes: debt_ratio <= 0.35 ?
│        ├── yes: predict "repaid"
│        └── no:  predict "default"
└── no:  predict "repaid"
```

Each leaf predicts the majority class of the training rows that ended up there (or their mean, for regression trees).

## How a tree is grown

Training is greedy. At each node, the algorithm tries every feature and every candidate threshold, and picks the split that makes the two child groups as **pure** as possible. Purity is usually measured with **Gini impurity**:

```text
gini = 1 - sum over classes of p_k^2
```

A pure node (all one class) has gini 0. A 50/50 node of two classes has gini `1 - (0.5^2 + 0.5^2) = 0.5`.

### Worked example: choosing a split

Ten rows, labels `[0,0,0,0,1,1,1,1,1,1]` sorted by feature `x`. The parent gini is `1 - (0.4^2 + 0.6^2) = 0.48`.

Candidate split after the 4th row (left = four 0s, right = six 1s):

```text
left gini = 0, right gini = 0
weighted = (4/10)*0 + (6/10)*0 = 0      -> perfect split
```

Candidate split after the 2nd row (left = two 0s, right = two 0s and six 1s):

```text
right gini = 1 - (0.25^2 + 0.75^2) = 0.375
weighted = (2/10)*0 + (8/10)*0.375 = 0.30
```

The first split wins. Candidate thresholds are the midpoints between consecutive sorted unique values. The tree then repeats the search inside each child until it reaches a stopping rule.

## Overfitting and depth

Left alone, a tree keeps splitting until every leaf is pure, which memorises the training data, noise included. You control this with:

- `max_depth`: the maximum number of questions from root to leaf.
- `min_samples_leaf`: the smallest allowed leaf, such as 20 rows.

A single deep tree has **high variance**: change a few training rows and the tree can look completely different.

## Random forests: many trees, one vote

A random forest reduces that variance by averaging many different trees:

1. **Bootstrap**: each tree trains on a random sample of rows drawn **with replacement** (the same size as the training set, so about 63% of distinct rows appear).
2. **Random features**: at each split, the tree may only consider a random subset of features (often the square root of the feature count). This stops every tree from splitting on the same strong feature first.
3. **Vote**: for classification, each tree predicts and the majority wins (or the probabilities are averaged).

Individual trees are noisy, but their errors are partly independent, so the average is much more stable. Rows left out of a tree's bootstrap sample (**out-of-bag** rows) give a free validation estimate.

## In scikit-learn

```python
from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(
    n_estimators=300, max_depth=None, min_samples_leaf=2,
    max_features="sqrt", oob_score=True, random_state=0)
forest.fit(X_train, y_train)
print(forest.oob_score_)                 # out-of-bag accuracy
print(forest.feature_importances_)       # average impurity decrease per feature
```

Trees do not need feature scaling, because a threshold on `income` works the same in dollars or thousands. Impurity-based feature importances are biased toward features with many unique values; **permutation importance** (`sklearn.inspection.permutation_importance`) is a more trustworthy check.

### Boosting, briefly

**Gradient boosting** (XGBoost, LightGBM, `HistGradientBoostingClassifier`) also combines trees, but builds them one after another, each fixing the previous ones' errors. It often wins on tabular benchmarks but needs more tuning. Random forests are the easier, safer first ensemble.

## Common mistakes

- **Growing one deep tree and trusting its training accuracy.** It will be close to 100% and meaningless.
- **Too few trees.** Results vary with the seed; more trees never overfit, they only cost time.
- **Reading feature importance as cause.**
- **Ordinal-encoding categories without thinking.** Trees can split on arbitrary integer codes, but a meaningless order can need extra splits (see Lesson 10).

## Quick recap

- A tree asks greedy yes/no questions that minimise weighted Gini impurity.
- Depth limits control overfitting; a single tree is high variance.
- A random forest = bootstrap samples + random feature subsets + majority vote.
- Start tabular problems with a logistic regression baseline and a random forest.

## Try it

On the Do tab, write Gini impurity, the best-threshold search and the majority vote, then train a small random forest from scratch.


Grow decision trees with Gini impurity and combine many of them into a random forest.

Trees and ensembles

Decision and Classification Trees, Clearly Explained!!!

StatQuest: Random Forests Part 1 - Building, Using and Evaluating

## Why this matters

A model is only as good as the number you use to judge it. Choose the wrong metric and you can ship a model that looks excellent and does nothing useful. This is most dangerous with **imbalanced classes**, which are the norm in practice: fraud, disease, equipment failure and churn are all rare. A metric must match what the model is for and what its mistakes cost.

## The accuracy trap

A disease affects 2% of 1,000 patients (20 people). A "model" that always says "healthy" is right 980 times: **98% accuracy**, and it never finds a single sick patient. Accuracy counts all correct answers equally, so the majority class dominates it.

## The confusion matrix

For a binary classifier, every prediction lands in one of four cells:

```text
                    predicted positive   predicted negative
actual positive          TP                   FN
actual negative          FP                   TN
```

- **TP** (true positive): sick, flagged. **FN** (false negative): sick, missed.
- **FP** (false positive): healthy, flagged. **TN** (true negative): healthy, cleared.

## Precision, recall and F1

```text
precision = TP / (TP + FP)    of the cases we flagged, how many were real?
recall    = TP / (TP + FN)    of the real cases, how many did we catch?
F1        = 2 * precision * recall / (precision + recall)
```

Recall is also called **sensitivity** or the true positive rate. F1 is the harmonic mean, which is high only when both are high.

### Worked example

A screening model on the 1,000 patients flags 50 people, 16 of whom are sick:

```text
TP = 16, FP = 34, FN = 4, TN = 946
precision = 16 / 50 = 0.32
recall    = 16 / 20 = 0.80
F1        = 2 * 0.32 * 0.80 / 1.12 ≈ 0.457
accuracy  = (16 + 946) / 1000 = 0.962
```

Accuracy went **down** compared with the useless "always healthy" model (0.962 vs 0.98), yet this model catches 80% of sick patients. For a cheap follow-up test, that trade is excellent. Precision and recall show it; accuracy hides it.

When the denominator is zero (the model flags nobody), precision is undefined. Libraries report 0 and warn; your own code should handle it too.

## Thresholds and the trade-off

A classifier outputs scores or probabilities, and the threshold decides who is flagged. Lowering the threshold flags more rows: recall rises, precision usually falls. There is no free lunch; you choose a point based on cost:

- **Recall first**: cancer screening, fraud review queues, safety alerts, where a miss is expensive.
- **Precision first**: spam filtering that deletes mail, automated account bans, where false alarms are expensive.

A practical recipe: "pick the threshold with the highest precision among those reaching at least 90% recall", chosen on the validation set.

## ROC and AUC

The **ROC curve** plots the true positive rate (recall) against the false positive rate `FP / (FP + TN)` as the threshold sweeps from high to low. The **area under it (AUC)** summarises ranking quality in one threshold-free number:

- AUC = 1.0: every positive scores above every negative.
- AUC = 0.5: no better than random ordering.

AUC has a useful interpretation: it is the probability that a randomly chosen positive scores higher than a randomly chosen negative (ties count half). That is exactly how you will compute it from scratch:

```python
import numpy as np

pos, neg = scores[y == 1], scores[y == 0]
greater = (pos[:, None] > neg[None, :]).mean()
ties = (pos[:, None] == neg[None, :]).mean()
auc = greater + 0.5 * ties
```

With heavy imbalance, ROC AUC can look comfortable while precision is poor, because the false positive rate divides by the huge negative count. The **precision-recall curve** and its average precision (`average_precision_score`) are more informative when positives are rare.

## In scikit-learn

```python
from sklearn.metrics import (confusion_matrix, precision_score, recall_score,
                             f1_score, roc_auc_score, average_precision_score,
                             classification_report)

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
print(roc_auc_score(y_test, proba), average_precision_score(y_test, proba))
```

For regression, the common metrics are **MAE** (mean absolute error, in label units, robust to outliers), **RMSE** (punishes big errors more) and **R-squared**.

## Common mistakes

- **Reporting accuracy alone on imbalanced data.** Always show the baseline and the confusion matrix.
- **Choosing the threshold on the test set.** Choose it on validation data.
- **Computing AUC from hard 0/1 predictions** instead of scores, which throws away the ranking.
- **Averaging metrics carelessly** across classes: macro averaging treats classes equally, micro averaging weights by count.

## Quick recap

- Accuracy misleads when classes are imbalanced.
- Precision = how trustworthy the flags are; recall = how many real cases are caught; F1 balances them.
- Thresholds trade precision for recall; set them by cost on validation data.
- ROC AUC measures ranking; prefer precision-recall metrics when positives are rare.

## Try it

On the Do tab, compute the confusion matrix, precision, recall, F1 and ROC AUC from scratch, and see why accuracy fools you.


Use the confusion matrix, precision, recall, F1 and ROC AUC to judge classifiers when classes are imbalanced.

Evaluation metrics

Machine Learning Fundamentals: The Confusion Matrix

ROC and AUC, Clearly Explained!

Linear and logistic regression, decision trees and random forests, and the metrics that show whether a classifier is any good.