Neural network basics

Why this matters

Linear and logistic regression draw straight boundaries. Many real problems need curved ones: the pixels of a handwritten digit, the words in a sentence, the interaction between two sensor readings. A neural network learns those curved boundaries by stacking simple layers. The same few ideas (layers, activations, a loss, backpropagation and gradient descent) power everything from a two-layer model on a spreadsheet to a large language model. In this lesson you build the smallest useful network by hand so the frameworks in the next lessons stop being magic.

The core idea: layers of weighted sums

A single neuron takes inputs x, computes a weighted sum plus a bias, then applies a non-linear activation function:

z = w1*x1 + w2*x2 + ... + b
a = activation(z)

A layer is many neurons side by side, which is one matrix multiplication. A two-layer network for binary classification looks like this:

hidden:  h = tanh(X @ W1 + b1)        X is (n, d), W1 is (d, k), h is (n, k)
output:  p = sigmoid(h @ W2 + b2)     W2 is (k,), p is (n,), each p in (0, 1)

The hidden layer turns each row into k new features, and the output layer is just logistic regression on those learned features. That is the key insight: the network learns its own features.

Why the activation? Without it, two layers collapse into one: (X @ W1) @ W2 equals X @ (W1 @ W2), still a straight line. tanh, sigmoid and ReLU (max(0, z)) bend the space so the next layer can separate classes a line cannot.

The loss and the backward pass

For binary labels we use binary cross-entropy, the same loss as logistic regression:

loss = -mean( y*log(p) + (1-y)*log(1-p) )

Training means nudging every weight against its gradient. Backpropagation is the chain rule applied layer by layer, from the output back to the input. For the network above the gradients are:

dz2 = (p - y) / n                    error signal at the output, shape (n,)
dW2 = h.T @ dz2                      db2 = sum(dz2)
dh  = outer(dz2, W2) * (1 - h**2)    (1 - h**2) is the derivative of tanh
dW1 = X.T @ dh                       db1 = dh.sum(axis=0)

The (p - y) term is neat: with sigmoid plus cross-entropy, the derivative simplifies to "prediction minus truth". Each layer multiplies the incoming error by its local derivative and passes it back.

Worked example: XOR

XOR is the classic problem a straight line cannot solve. The label is 1 when two inputs have the same sign:

Logistic regression tops out near 50% accuracy here. A network with 8 hidden tanh units learns to combine several tilted lines into a checkerboard boundary and reaches above 95%. A full training loop in NumPy is short:

for epoch in range(3000):
    h, p = forward(X, params)          # forward pass
    grads = backward(X, y, h, p, params)
    for k in params:                   # gradient descent step
        params[k] -= lr * grads[k]

Checking your gradients

Backprop bugs are silent: the code runs and the loss just falls slowly or not at all. The standard defence is a numerical gradient check. Nudge one weight by a tiny eps up and down, measure the loss each time, and compare:

numerical = (loss(w + eps) - loss(w - eps)) / (2 * eps)

If that matches your analytic gradient to about six decimal places, your backward pass is right. Frameworks such as PyTorch compute gradients automatically (autograd), so you rarely do this at work, but it is the best way to understand what autograd does.

Initialisation matters

If every weight starts at zero, every hidden unit computes the same thing and receives the same gradient, so they never become different. This is the symmetry problem. Networks start from small random weights, usually scaled by the layer size (Xavier/Glorot or He initialisation). Fix the random seed while you learn so your results are reproducible.

Common mistakes

• Forgetting the activation. Stacked linear layers are still linear.
• Shape errors. Write the shape of every array in a comment, as above. Most bugs are a missing transpose.
• Learning rate too high. The loss jumps around or becomes nan. Halve it until the loss falls smoothly.
• Unscaled inputs. Features on very different scales make training slow. Standardise inputs first (mean 0, standard deviation 1).
• Judging by training accuracy alone. A network can memorise training data. The next lesson covers validation curves.

Quick recap

• A layer is a matrix multiplication plus bias followed by a non-linear activation.
• Hidden layers learn features; the output layer turns them into a prediction.
• Backpropagation is the chain rule, layer by layer, from the loss back to each weight.
• Gradient checks catch backprop bugs; random initialisation breaks symmetry.

Try it

Open the Do tab. You will write the forward and backward passes of a two-layer network in NumPy, pass a numerical gradient check, and train it to solve XOR.