The calculus and linear algebra behind science, engineering and machine learning: derivatives, integrals, vectors, matrices, gradients and eigenvalues, with Python exercises.

Calculus & Linear Algebra for STEM

## Why this matters

Almost every quantity a scientist or engineer cares about is a rate: how fast a reaction uses up a reagent, how quickly a beam bends as load increases, how fast a model's error falls during training, how a premium changes as age goes up. Calculus is the toolkit for rates, and it starts with one idea: the **limit**.

You do not need to be a math whiz to use it. In practice you will compute most rates numerically with a few lines of code, and the formal rules in the next lesson are shortcuts for what the code does by brute force.

## Average rate of change

Suppose a tank's water level (in centimetres) after t minutes is h(t) = t² + 2t. Between t = 1 and t = 3:

- h(1) = 1 + 2 = 3
- h(3) = 9 + 6 = 15
- average rate = (15 − 3) / (3 − 1) = 6 cm per minute

That is the slope of the straight line (the **secant line**) joining two points on the curve. The general formula is the **difference quotient**:

```text
average rate on [x, x + h] = (f(x + h) − f(x)) / h
```

## From average to instantaneous

What is the rate *exactly* at t = 1? A single instant has no width, so we cannot divide by zero. Instead we shrink the interval and watch what the average rate approaches:

| h | (h(1 + h) − h(1)) / h |
|---|---|
| 1 | 5 |
| 0.1 | 4.1 |
| 0.01 | 4.01 |
| 0.001 | 4.001 |

The values head toward 4. We write this as a **limit**:

```text
lim (h → 0) of (f(x + h) − f(x)) / h = 4    at x = 1
```

Read it as: "as h gets closer and closer to 0, the quotient gets as close as we like to 4". The limit is the slope of the **tangent line**, the instantaneous rate of change. It has a name you will use for the rest of this course: the **derivative**, written f′(x).

## What a limit really says

A limit describes where a function is *heading*, not necessarily its value at that point. A classic example:

```text
g(x) = (x² − 1) / (x − 1)
```

At x = 1 the formula gives 0/0, which is undefined. But for every other x, g(x) = x + 1, so as x approaches 1, g(x) approaches 2. The limit is 2 even though g(1) does not exist. Difference quotients always look like 0/0 at h = 0; the limit is what rescues them.

A limit fails to exist when the function jumps (different values from the left and right) or blows up toward infinity. Smooth physical quantities, such as position, temperature and concentration, usually have well-behaved limits, which is why calculus works so well in science.

## Estimating a rate numerically

Code can do the shrinking for you. This is how many simulation and optimisation libraries compute derivatives when no formula is available:

```python
def f(x):
    return x ** 2 + 2 * x

def forward_difference(f, x, h=1e-6):
    return (f(x + h) - f(x)) / h

print(forward_difference(f, 1))   # about 4.000001
```

A more accurate version looks both ways from x, which cancels most of the error:

```python
def central_difference(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)
```

## Common mistakes

- **Making h too small.** Computers store numbers with about 16 significant digits. With h = 1e-15 the subtraction f(x + h) − f(x) loses almost all its digits and the answer becomes noise. Values between 1e-4 and 1e-8 are a safe range.
- **Confusing average and instantaneous rates.** "The car averaged 60 km/h" does not mean the speedometer read 60 at 3:15 pm.
- **Plugging in first.** Substituting h = 0 into a difference quotient gives 0/0. Simplify or estimate first, then take the limit.
- **Forgetting units.** A rate has units of output per input: cm per minute, dollars per year, error per training step.

## Quick recap

- Average rate = (f(x + h) − f(x)) / h, the slope of a secant line.
- The instantaneous rate is the limit of that quotient as h → 0: the derivative.
- A limit is where a function is heading; the function need not be defined there.
- Numerically, a small h (about 1e-6) gives a good estimate; the central difference is more accurate.

## Try it

Open the **Do** tab and write the difference quotient that estimates a rate of change, then use it on a few functions.


Go from the average rate of change between two points to the instantaneous rate at one point, and estimate it with code.

Limits and rates of change

Introduction to limits | Limits | Differential Calculus | Khan Academy

The paradox of the derivative | Chapter 2, Essence of calculus

## Why this matters

Estimating a derivative numerically works, but a formula is faster, exact and tells you more. If a cost model is C(q) = 0.02q³ − 3q² + 200q, you want C′(q), the marginal cost, as a formula you can reason about, not a table of estimates. A handful of rules covers nearly every function you will meet at work.

## Notation

The derivative of f is written f′(x) or df/dx. Read df/dx as "the small change in f per small change in x". Both mean the same thing: the slope of f at x.

## The core rules

| Rule | Formula | Example |
|---|---|---|
| Constant | (c)′ = 0 | (7)′ = 0 |
| Power | (xⁿ)′ = n·xⁿ⁻¹ | (x⁵)′ = 5x⁴ |
| Constant multiple | (c·f)′ = c·f′ | (3x²)′ = 6x |
| Sum | (f + g)′ = f′ + g′ | (x² + x)′ = 2x + 1 |
| Product | (f·g)′ = f′·g + f·g′ | see below |
| Chain | (f(g(x)))′ = f′(g(x))·g′(x) | see below |

A few standard derivatives are worth memorising: (eˣ)′ = eˣ, (ln x)′ = 1/x, (sin x)′ = cos x, (cos x)′ = −sin x.

## Polynomials: power rule plus sum rule

Differentiate term by term. Each term a·xⁿ becomes n·a·xⁿ⁻¹:

```text
f(x)  = 4x³ − 5x² + 2x − 9
f′(x) = 12x² − 10x + 2
```

The constant −9 disappears because a constant does not change. The marginal cost from the opening example is C′(q) = 0.06q² − 6q + 200.

A polynomial is easy to handle in code as a list of coefficients, lowest power first. 4x³ − 5x² + 2x − 9 is `[-9, 2, -5, 4]`: the coefficient at index k belongs to xᵏ. The power rule then says: the new coefficient at index k − 1 is k times the old one at index k.

## The product rule

For a product, you cannot just multiply the two derivatives. Take f(x) = x²·(3x + 1):

```text
u = x²       u′ = 2x
v = 3x + 1   v′ = 3
f′ = u′·v + u·v′ = 2x(3x + 1) + x²·3 = 9x² + 2x
```

Check by expanding first: f(x) = 3x³ + x², so f′(x) = 9x² + 2x. Same answer. Multiplying the derivatives (2x · 3 = 6x) would have been wrong.

## The chain rule

Use it for a function inside a function, such as f(x) = (2x + 1)⁵. Differentiate the outside, keep the inside, then multiply by the derivative of the inside:

```text
outer: u⁵ → 5u⁴
inner: u = 2x + 1 → u′ = 2
f′(x) = 5(2x + 1)⁴ · 2 = 10(2x + 1)⁴
```

The chain rule is everywhere in applied work. Exponential decay e^(−kt) has derivative −k·e^(−kt). Training a neural network (backpropagation) is the chain rule applied through many layers.

## Checking your work with code

A quick numerical check catches most algebra slips:

```python
def f(x):
    return (2 * x + 1) ** 5

def f_prime(x):
    return 10 * (2 * x + 1) ** 4

h = 1e-5
x = 0.3
estimate = (f(x + h) - f(x - h)) / (2 * h)
print(estimate, f_prime(x))   # the two numbers should agree closely
```

Libraries such as SymPy (symbolic algebra) can also differentiate formulas for you. Knowing the rules means you can read and sanity-check what they return.

## Common mistakes

- **Forgetting to lower the power.** (x³)′ is 3x², not 3x³.
- **Multiplying derivatives for a product.** (f·g)′ is f′g + fg′, never f′·g′.
- **Dropping the inner derivative in the chain rule.** ((3x)²)′ = 2(3x)·3 = 18x, not 6x.
- **Keeping constants.** The derivative of a plain number is 0.

## Quick recap

- Power rule: bring the exponent down and subtract one.
- Differentiate sums term by term; constants vanish.
- Products need the product rule; nested functions need the chain rule.
- A central-difference check in code is a cheap way to confirm a hand derivation.

## Try it

In the **Do** tab, write the power rule for a polynomial stored as a coefficient list, then watch the code confirm the product rule.


Differentiate polynomials, products and nested functions with the power, product and chain rules.

Derivative rules

The Power Rule For Derivatives

Visualizing the chain rule and product rule | Chapter 4, Essence of calculus

## Why this matters

Engineers minimise material and weight, chemists maximise yield, operations researchers minimise cost, and machine learning minimises error. All of these are **optimisation** problems, and the derivative is the first tool for solving them: at the top of a hill or the bottom of a valley, the slope is zero.

## The core idea

If f is smooth, a maximum or minimum inside an interval happens where f′(x) = 0. Those points are called **critical points**. To find the best value:

1. Write the quantity you care about as a function of one variable.
2. Differentiate it.
3. Solve f′(x) = 0 for the critical points.
4. Check each one, and the endpoints of the allowed interval, to see which gives the largest or smallest value.

To tell a maximum from a minimum, look at the sign of f′ on either side (positive then negative means a peak) or use the **second derivative**: f″(x) < 0 means the curve bends downward (a maximum), f″(x) > 0 means it bends upward (a minimum).

## Worked example: the open box

You have a 30 cm × 30 cm sheet of metal. Cut a square of side x from each corner and fold up the sides to make an open box. What x gives the largest volume?

The base is (30 − 2x) on each side and the height is x, so:

```text
V(x) = x(30 − 2x)²        for 0 < x < 15
```

Expand and differentiate:

```text
V(x)  = 4x³ − 120x² + 900x
V′(x) = 12x² − 240x + 900 = 12(x² − 20x + 75) = 12(x − 5)(x − 15)
```

V′(x) = 0 at x = 5 and x = 15. At x = 15 the base has zero width (V = 0), so the candidate is x = 5. Check the second derivative: V″(x) = 24x − 240, and V″(5) = −120 < 0, so x = 5 is a maximum. The best box has volume 5 · 20² = 2,000 cm³.

## When you cannot solve f′(x) = 0 by hand

Real models are rarely this tidy. Numerically, you can search for the point where the slope changes sign. **Bisection** is the simplest reliable method: if the slope is positive at the left end of an interval and negative at the right end, a peak lies between them. Check the midpoint; if the slope there is still positive, the peak is to the right, so move the left end up; otherwise move the right end down. Each step halves the interval.

```python
def slope(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2 * h)

lo, hi = 0.01, 7.5
for _ in range(60):
    mid = (lo + hi) / 2
    if slope(volume, mid) > 0:
        lo = mid
    else:
        hi = mid
```

After 60 halvings the interval is far smaller than anything you could measure. Library optimisers such as `scipy.optimize.minimize_scalar` use smarter versions of the same idea, and gradient descent (Module 4) extends it to many variables.

## Related rates, briefly

The derivative also links rates that change together. If a spherical balloon's radius grows at 2 cm/s, its volume V = (4/3)πr³ grows at dV/dt = 4πr² · dr/dt. At r = 10 cm that is 4π · 100 · 2 ≈ 2,513 cm³/s. The chain rule does the work: differentiate the relationship with respect to time.

## Common mistakes

- **Forgetting the endpoints.** On a closed interval, the best value can sit at an edge where f′ is not zero.
- **Assuming every critical point is a maximum.** Check the second derivative or the sign change. Some critical points (like x³ at 0) are neither.
- **Ignoring the physical domain.** A negative length or a cut wider than half the sheet is not a real answer.
- **Optimising the wrong quantity.** Write down exactly what you are maximising, with units, before differentiating.

## Quick recap

- Maxima and minima of a smooth function occur where f′(x) = 0 or at the ends of the interval.
- The second derivative tells a peak (f″ < 0) from a valley (f″ > 0).
- Bisection on the sign of the slope finds an optimum numerically.
- Related rates use the chain rule to connect rates of change over time.

## Try it

In the **Do** tab, finish the bisection step that finds the best box size numerically.


Find the maximum or minimum of a quantity by setting its derivative to zero, by hand and numerically.

Optimisation with derivatives

Optimization with Calculus 1

Optimization: box volume (Part 1) | Applications of derivatives | AP Calculus AB | Khan Academy

Limits, derivative rules and optimisation: how to measure and use rates of change, by hand and in code.