Images

Why this matters

Images show up in quality inspection, medical imaging, document processing, retail and agriculture. A 224 by 224 colour photo is about 150,000 numbers, and the meaning (a crack, a tumour, a ripe apple) depends on local patterns wherever they appear. Convolutional neural networks (CNNs) are built for exactly that, and transfer learning lets you reuse a network trained on millions of images so you can solve your own problem with a few hundred labelled examples and a free GPU.

Why not a plain network?

Flatten an image into one long vector and feed it to a dense layer, and two things go wrong. First, the parameter count explodes: 150,000 inputs times 1,000 hidden units is 150 million weights in the first layer. Second, the network has to learn "edge" separately at every position, because nothing tells it that a pattern in the top-left corner is the same pattern in the bottom-right.

The core idea: convolution

A convolution slides a small filter (say 3 by 3) across the image and, at each position, computes a weighted sum of the pixels under it. The same nine weights are reused everywhere (weight sharing), so the layer has few parameters and detects its pattern wherever it appears.

Worked example: a vertical-edge filter on a tiny greyscale image.

image patch         filter            result
10 10  0            1  0 -1
10 10  0     *      1  0 -1     =  (10+10+10) - (0+0+0) = 30   strong edge
10 10  0            1  0 -1

 5  5  5            1  0 -1
 5  5  5     *      1  0 -1     =  15 - 15 = 0                  no edge
 5  5  5            1  0 -1

A convolutional layer learns many such filters at once (32, 64, 128...). Each produces a feature map showing where its pattern occurs.

Building blocks of a CNN

• Convolution + ReLU: detect local patterns.
• Pooling (usually max pooling, 2 by 2): keep the strongest response in each small window, halving width and height. This adds a little tolerance to small shifts and cuts computation.
• Stacking: early layers find edges and colours; middle layers combine them into textures and parts; deep layers respond to whole objects. Each layer sees a larger region of the original image (its receptive field grows).
• Head: global average pooling, then a dense layer that outputs class scores.

Well-known architectures you will see in libraries: ResNet (skip connections that let very deep networks train), EfficientNet and MobileNet (accurate and small, good for phones), and Vision Transformers (ViT), which split an image into patches and use attention instead of convolution. For most applied work, a pretrained ResNet or EfficientNet is a strong start.

Transfer learning

Training a CNN from scratch needs a lot of labelled images and compute. Instead:

1. Take a model pretrained on a large dataset such as ImageNet (1.2 million images, 1,000 classes). Its early and middle layers already detect general visual features.
2. Replace the head with a new final layer for your classes.
3. Feature extraction: freeze the pretrained layers and train only the new head. Fast, works with small datasets.
4. Fine-tuning (optional): unfreeze some later layers and train them with a small learning rate (for example 10 times lower), so the features adapt to your images without being destroyed.

In PyTorch with torchvision:

import torch
from torch import nn
from torchvision import models

weights = models.ResNet18_Weights.DEFAULT
model = models.resnet18(weights=weights)
for p in model.parameters():
    p.requires_grad = False                          # freeze the backbone
model.fc = nn.Linear(model.fc.in_features, 3)        # new head: 3 classes
optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)
preprocess = weights.transforms()                    # resize + normalise like training

In Keras the same idea uses keras.applications.EfficientNetB0(include_top=False, weights="imagenet") with base.trainable = False and a new Dense layer on top.

Data augmentation

Small image datasets overfit quickly. Augmentation creates new training variants on the fly: random crops, horizontal flips, small rotations, colour jitter. Choose augmentations that preserve the label. Flipping a cat photo is fine; flipping a road sign with an arrow, or a chest X-ray where left and right matter, is not. Apply augmentation to training data only.

Practical checklist

• Use the same preprocessing as the pretrained model (input size, normalisation means and standard deviations), or accuracy collapses.
• Split by source, not by image: if one patient, product batch or camera has many photos, keep them all in one split to avoid leakage.
• Look at the errors. A confusion matrix plus a grid of misclassified images often reveals label mistakes or a shortcut (the model learned the watermark, not the defect).
• Mind the licence and consent for any image dataset, especially photos of people.

Common mistakes

• Training from scratch on a few hundred images.
• Fine-tuning everything with a high learning rate, which wipes out the pretrained features.
• Label-changing augmentations.
• Forgetting model.eval() at inference, which leaves batch-normalisation layers in training mode and changes predictions.

Quick recap

• Convolutions reuse small filters across the image, so CNNs detect patterns anywhere with few parameters.
• Layers build from edges to parts to objects; pooling shrinks the maps.
• Transfer learning: pretrained backbone, new head, train the head, then optionally fine-tune with a small learning rate.
• Augment training data with label-preserving transforms and split by source to avoid leakage.