Go from classic ML to the models used at work: neural networks, embeddings, CNNs, gradient boosting and LLM apps, plus evaluation, experiment tracking and a capstone.

Applied Machine Learning

## Why this matters

Support tickets, product reviews, clinical notes, contracts: a large share of business data is text. Models cannot read words directly; they need numbers. **Embeddings** turn text into vectors so that similar meanings land close together. Once text is a vector, everything you already know applies: nearest neighbours, logistic regression, clustering, search. Most practical text classification today is "embed the text, then train a small model on top".

## From words to vectors

The oldest approach is **bag of words**: one column per vocabulary word, counting how often it appears (often weighted by TF-IDF so common words count less). It works well for simple tasks but treats "refund" and "reimbursement" as unrelated columns.

**Word embeddings** (word2vec, GloVe) fix that. Each word gets a dense vector of, say, 300 numbers, learned from billions of sentences so that words used in similar contexts get similar vectors. Famous example: `vector("king") - vector("man") + vector("woman")` lands near `vector("queen")`.

**Sentence embeddings** go one step further: a transformer model reads the whole sentence, word order and all, and outputs one vector for it. These are what you will mostly use.

## Measuring similarity: cosine

To compare two vectors we use **cosine similarity**, the cosine of the angle between them:

```text
cosine(a, b) = (a . b) / (|a| * |b|)
```

It ranges from 1 (same direction) through 0 (unrelated) to -1 (opposite). Cosine ignores vector length, which is what you want: a long message and a short one about the same topic should still match.

```python
import numpy as np

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

cosine(np.array([1.0, 2.0]), np.array([2.0, 4.0]))   # 1.0, same direction
```

## Worked example: routing support tickets

You want to route messages into `billing`, `delivery`, `access` or `praise`. A simple but strong recipe:

1. **Embed** every training message. With word vectors, average the vectors of the words you know.
2. **Compute a centroid** per label: the mean embedding of its training messages.
3. **Predict** the label whose centroid has the highest cosine similarity to the new message.

```text
"Please refund the payment"  -> embed -> closest centroid: billing
"The package never arrived"  -> embed -> closest centroid: delivery
```

This **nearest-centroid** classifier needs only a handful of examples per class, which makes it a useful baseline and a good fit when labelled data is scarce. With more labels, train logistic regression on the embeddings instead.

## The tools you will use

**sentence-transformers** gives you a pretrained sentence embedding model in three lines (free, runs on a laptop CPU or in Colab):

```python
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

encoder = SentenceTransformer("all-MiniLM-L6-v2")      # small, 384-dimensional
X_train = encoder.encode(train_texts, normalize_embeddings=True)
X_test = encoder.encode(test_texts, normalize_embeddings=True)

clf = LogisticRegression(max_iter=1000).fit(X_train, train_labels)
print(clf.score(X_test, test_labels))
```

With `normalize_embeddings=True` every vector has length 1, so a plain dot product equals cosine similarity.

For a classic baseline, scikit-learn's `TfidfVectorizer` plus `LogisticRegression` is fast and surprisingly hard to beat on keyword-driven tasks. Always run it first: if TF-IDF already scores 95%, a bigger model may not be worth the cost.

For the best accuracy with thousands of labelled examples, **fine-tune** a small transformer (for example DistilBERT) with the Hugging Face `transformers` library. That updates the encoder itself for your labels, at the cost of a GPU and more care.

## Practical details

- **Preprocessing:** with modern embeddings, keep the text mostly as written. Lowercasing and stripping punctuation help bag-of-words, but transformer models were trained on natural text.
- **Long texts:** embedding models have a maximum input length (often 256–512 tokens). Truncate, or split into chunks and average.
- **Language and domain:** a general English model may do poorly on legal, medical or non-English text. Check a domain or multilingual model.
- **Class imbalance:** if 90% of tickets are billing, accuracy is misleading. Report per-class precision and recall (from Machine Learning Foundations).
- **Privacy:** if you send text to a hosted embedding API, you are sharing it with a third party. Local models avoid that.

## Common mistakes

- **Comparing vectors from different models.** Embeddings from two models live in different spaces; never mix them.
- **Euclidean distance on unnormalised embeddings** when the model was trained for cosine.
- **Leakage through duplicates.** Near-identical tickets in train and test inflate the score. Deduplicate before splitting.
- **Skipping the baseline.** Always compare with TF-IDF + logistic regression and with "predict the most common class".

## Quick recap

- Embeddings map text to vectors where similar meaning means a small angle.
- Cosine similarity compares direction and ignores length.
- Embed, then classify: nearest centroid for few labels, logistic regression for more, fine-tuning for the most.
- Start with a TF-IDF baseline and check per-class metrics.

## Try it

Open the **Do** tab. Using a small embedding table, you will write `embed`, `cosine`, `fit_centroids` and `predict`, and route five support messages to the right team.


Turn text into embedding vectors, compare them with cosine similarity and classify support messages.

Text and embeddings

What are Word Embeddings?

Word Embedding and Word2Vec, Clearly Explained!!!

## Why this matters

Images show up in quality inspection, medical imaging, document processing, retail and agriculture. A 224 by 224 colour photo is about 150,000 numbers, and the meaning (a crack, a tumour, a ripe apple) depends on local patterns wherever they appear. **Convolutional neural networks (CNNs)** are built for exactly that, and **transfer learning** lets you reuse a network trained on millions of images so you can solve your own problem with a few hundred labelled examples and a free GPU.

## Why not a plain network?

Flatten an image into one long vector and feed it to a dense layer, and two things go wrong. First, the parameter count explodes: 150,000 inputs times 1,000 hidden units is 150 million weights in the first layer. Second, the network has to learn "edge" separately at every position, because nothing tells it that a pattern in the top-left corner is the same pattern in the bottom-right.

## The core idea: convolution

A **convolution** slides a small filter (say 3 by 3) across the image and, at each position, computes a weighted sum of the pixels under it. The same nine weights are reused everywhere (**weight sharing**), so the layer has few parameters and detects its pattern wherever it appears.

Worked example: a vertical-edge filter on a tiny greyscale image.

```text
image patch         filter            result
10 10  0            1  0 -1
10 10  0     *      1  0 -1     =  (10+10+10) - (0+0+0) = 30   strong edge
10 10  0            1  0 -1

 5  5  5            1  0 -1
 5  5  5     *      1  0 -1     =  15 - 15 = 0                  no edge
 5  5  5            1  0 -1
```

A convolutional layer learns many such filters at once (32, 64, 128...). Each produces a **feature map** showing where its pattern occurs.

## Building blocks of a CNN

- **Convolution + ReLU:** detect local patterns.
- **Pooling** (usually max pooling, 2 by 2): keep the strongest response in each small window, halving width and height. This adds a little tolerance to small shifts and cuts computation.
- **Stacking:** early layers find edges and colours; middle layers combine them into textures and parts; deep layers respond to whole objects. Each layer sees a larger region of the original image (its **receptive field** grows).
- **Head:** global average pooling, then a dense layer that outputs class scores.

Well-known architectures you will see in libraries: **ResNet** (skip connections that let very deep networks train), **EfficientNet** and **MobileNet** (accurate and small, good for phones), and **Vision Transformers (ViT)**, which split an image into patches and use attention instead of convolution. For most applied work, a pretrained ResNet or EfficientNet is a strong start.

## Transfer learning

Training a CNN from scratch needs a lot of labelled images and compute. Instead:

1. Take a model **pretrained** on a large dataset such as ImageNet (1.2 million images, 1,000 classes). Its early and middle layers already detect general visual features.
2. **Replace the head** with a new final layer for your classes.
3. **Feature extraction:** freeze the pretrained layers and train only the new head. Fast, works with small datasets.
4. **Fine-tuning (optional):** unfreeze some later layers and train them with a small learning rate (for example 10 times lower), so the features adapt to your images without being destroyed.

In PyTorch with torchvision:

```python
import torch
from torch import nn
from torchvision import models

weights = models.ResNet18_Weights.DEFAULT
model = models.resnet18(weights=weights)
for p in model.parameters():
    p.requires_grad = False                          # freeze the backbone
model.fc = nn.Linear(model.fc.in_features, 3)        # new head: 3 classes
optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)
preprocess = weights.transforms()                    # resize + normalise like training
```

In Keras the same idea uses `keras.applications.EfficientNetB0(include_top=False, weights="imagenet")` with `base.trainable = False` and a new `Dense` layer on top.

## Data augmentation

Small image datasets overfit quickly. **Augmentation** creates new training variants on the fly: random crops, horizontal flips, small rotations, colour jitter. Choose augmentations that preserve the label. Flipping a cat photo is fine; flipping a road sign with an arrow, or a chest X-ray where left and right matter, is not. Apply augmentation to training data only.

## Practical checklist

- Use the **same preprocessing** as the pretrained model (input size, normalisation means and standard deviations), or accuracy collapses.
- Split by **source**, not by image: if one patient, product batch or camera has many photos, keep them all in one split to avoid leakage.
- Look at the errors. A confusion matrix plus a grid of misclassified images often reveals label mistakes or a shortcut (the model learned the watermark, not the defect).
- Mind the licence and consent for any image dataset, especially photos of people.

## Common mistakes

- **Training from scratch** on a few hundred images.
- **Fine-tuning everything with a high learning rate**, which wipes out the pretrained features.
- **Label-changing augmentations.**
- **Forgetting `model.eval()`** at inference, which leaves batch-normalisation layers in training mode and changes predictions.

## Quick recap

- Convolutions reuse small filters across the image, so CNNs detect patterns anywhere with few parameters.
- Layers build from edges to parts to objects; pooling shrinks the maps.
- Transfer learning: pretrained backbone, new head, train the head, then optionally fine-tune with a small learning rate.
- Augment training data with label-preserving transforms and split by source to avoid leakage.


Understand how convolutional networks find patterns in images and how transfer learning reuses pretrained models.

Images

What are Convolutional Neural Networks (CNNs)?

Transfer Learning (C3W2L07)

## Why this matters

Neural networks win on images, audio and text. On **tabular data** (rows and columns in a database: customers, transactions, sensor logs, insurance claims) the usual winner is **gradient boosting**: libraries such as XGBoost, LightGBM and CatBoost, or scikit-learn's `HistGradientBoostingClassifier`. Boosted trees handle mixed feature types, missing values and non-linear interactions with little preprocessing, and they train in seconds on a laptop. If you work with business data, this is likely the model you will ship most often.

## The core idea: fix the previous mistakes

A random forest (from Machine Learning Foundations) trains many deep trees **independently** and averages them. Boosting trains many small trees **one after another**, each one correcting the errors of the ensemble so far.

For regression with squared error, the algorithm is:

```text
F0(x) = mean(y)                               start with a constant
for m = 1..M:
    r = y - F(x)                              residuals: what is still wrong
    fit a small tree h_m to predict r
    F(x) = F(x) + learning_rate * h_m(x)      take a small step toward fixing it
```

The residual `y - F(x)` is the negative gradient of the squared-error loss with respect to the prediction. That is where the "gradient" in gradient boosting comes from: each tree is a step of gradient descent, taken in function space. For other losses (log loss for classification, absolute error for robust regression) you fit the tree to that loss's negative gradient instead.

## Worked example by hand

Three houses, one feature (size), target price in thousands:

| size | price | F0 | residual |
|---|---|---|---|
| 50 | 100 | 200 | -100 |
| 80 | 180 | 200 | -20 |
| 120 | 320 | 200 | +120 |

`F0` is the mean, 200, and the mean squared error is (100² + 20² + 120²)/3 ≈ 8,267. The best single split for the residuals is at size 100: left mean -60, right mean +120. With learning rate 0.5:

```text
house 1: 200 + 0.5 * (-60)  = 170
house 2: 200 + 0.5 * (-60)  = 170
house 3: 200 + 0.5 * (+120) = 260
```

The mean squared error fell to (70² + 10² + 60²)/3 ≈ 2,867. The next tree is fitted to the new residuals (-70, +10, +60), and so on. Each round removes part of the remaining error, and the small learning rate stops any single tree from overreacting.

## The settings that matter

- **`n_estimators`** (number of trees) and **`learning_rate`** work together. A smaller rate (0.05–0.1) needs more trees but usually generalises better. Use early stopping on a validation set to pick the number of trees.
- **Tree size** (`max_depth` 3–8, or `num_leaves` in LightGBM): depth 1 trees (stumps) model each feature separately; deeper trees capture interactions between features.
- **Subsampling** (`subsample`, `colsample_bytree` around 0.8): train each tree on a random part of the rows or columns. This adds randomness that reduces overfitting.
- **Regularisation** (`min_child_weight`, `reg_lambda`): stop leaves from fitting a handful of rows.

## In code

scikit-learn ships a fast histogram-based implementation:

```python
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score

model = HistGradientBoostingClassifier(
    learning_rate=0.05, max_iter=500, max_depth=6,
    early_stopping=True, validation_fraction=0.15, random_state=0,
)
model.fit(X_train, y_train)
print(roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]))
```

XGBoost has a similar interface:

```python
import xgboost as xgb

model = xgb.XGBClassifier(n_estimators=2000, learning_rate=0.05, max_depth=6,
                          subsample=0.8, colsample_bytree=0.8,
                          early_stopping_rounds=50, eval_metric="auc")
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
print(model.best_iteration)
```

LightGBM and CatBoost follow the same pattern. CatBoost handles categorical columns natively; LightGBM is often the fastest on large data. Install them with `pip`; all three work in free Colab and Kaggle notebooks.

## Interpreting a boosted model

- **Feature importance** (gain or split counts) is a quick overview but can mislead with correlated features.
- **Permutation importance** (`sklearn.inspection.permutation_importance`) measures how much the validation score drops when one column is shuffled.
- **SHAP values** (the `shap` library) explain individual predictions: how much each feature pushed this row's prediction up or down.

## Common mistakes

- **Too many trees without early stopping.** Boosting keeps reducing training error forever and eventually overfits.
- **Learning rate too high** (for example 1.0) with deep trees: fast to overfit.
- **Leaky features.** Trees exploit any column that encodes the answer, such as a "cancellation date" when predicting cancellations. Ask of each feature: would I know this at prediction time?
- **One-hot encoding high-cardinality IDs** into thousands of columns. Use native categorical support or target encoding inside cross-validation.
- **Skipping the baseline.** Compare with logistic regression and with predicting the mean.

## Quick recap

- Boosting adds small trees one at a time, each fitted to the current residuals (the negative gradient).
- The learning rate shrinks each step; tune it together with the number of trees, using early stopping.
- On tabular data, boosted trees are the default strong model; XGBoost, LightGBM, CatBoost and scikit-learn all implement it.
- Explain models with permutation importance or SHAP, and watch for leaky features.

## Try it

Open the **Do** tab. You will implement gradient boosting with decision stumps in NumPy, confirm that the training error falls every round, and beat a predict-the-mean baseline five times over.


Implement gradient boosting with decision stumps and learn how XGBoost, LightGBM and scikit-learn apply it to tabular data.

Gradient boosting

Gradient Boost Part 1 (of 4): Regression Main Ideas

XGBoost Part 1 (of 4): Regression

Apply ML to text with embeddings, to images with CNNs and transfer learning, and to tabular data with gradient boosting.