Use NumPy arrays, plots, fitting, numerical methods, units and instrument data files to do real engineering and science calculations in Python.

Scientific Python

## Why this matters

In 1999 NASA lost the Mars Climate Orbiter because one team's software reported thruster impulse in pound-force seconds while the navigation software expected newton seconds. Every number was computed correctly; the units were not. Smaller versions of that mistake happen every day: a pressure in kPa used as if it were Pa, a flow in L/min mixed with m³/s, a drawing in millimetres read as inches, a concentration in mg/L confused with mmol/L. Python's floats carry no units, so **you** have to.

## Strategy 1: convert to SI at the edges

The simplest reliable habit: convert every input to SI base units (m, kg, s, A, K, mol) as soon as it enters your code, compute in SI, and convert back only for output.

```python
MM = 1e-3          # metres per millimetre
KPA = 1e3          # pascals per kilopascal
MIN = 60.0         # seconds per minute
L = 1e-3           # cubic metres per litre

d = 25 * MM                    # pipe diameter, m
flow = 40 * L / MIN            # 40 L/min in m^3/s
area = 3.14159 * d**2 / 4      # m^2
velocity = flow / area         # m/s
print(f"{velocity:.2f} m/s")   # 1.36 m/s
```

Naming the constants (`MM`, `KPA`) makes the conversions readable and searchable. Put units in variable names or comments when they are not SI: `temp_C`, `length_mm`, `pressure_kPa`. Your future self will thank you.

**Temperature is special**: °C and K differ by an offset, not just a factor. A temperature *difference* of 10 °C is 10 K, but a temperature of 10 °C is 283.15 K. Formulas such as the ideal gas law and radiation need absolute temperatures.

## Strategy 2: dimensional analysis

Every physical quantity has dimensions built from length L, mass M, time T, current I, temperature Θ and amount N. Force is `M L T⁻²`; energy is `M L² T⁻²`; pressure is `M L⁻¹ T⁻²`. Three rules catch most unit errors:

1. You can only **add, subtract or compare** quantities with the **same dimensions**. Adding a length to a time is a bug.
2. **Multiplying and dividing** combine dimensions: exponents add and subtract.
3. Arguments of `exp`, `log` and `sin` must be **dimensionless**. In `e^(-t/τ)`, `t` and `τ` must share units.

You can check a formula on paper before you run it. For example, the kinetic energy `½ m v²` has dimensions `M (L T⁻¹)² = M L² T⁻²`, which is energy, as it should be.

### Worked example: a tiny dimension checker

Representing dimensions as exponent tuples lets code enforce those rules:

```python
class Q:
    def __init__(self, value, dims):       # dims = (L, M, T)
        self.value, self.dims = value, tuple(dims)

    def __mul__(self, other):
        return Q(self.value * other.value,
                 [a + b for a, b in zip(self.dims, other.dims)])

    def __add__(self, other):
        if self.dims != other.dims:
            raise ValueError(f"cannot add {self.dims} and {other.dims}")
        return Q(self.value + other.value, self.dims)

m = Q(1.0, (1, 0, 0)); s = Q(1.0, (0, 0, 1))
m + s        # ValueError: cannot add (1, 0, 0) and (0, 0, 1)
```

That is the core idea behind every units library.

## Strategy 3: a units library (pint)

For real work, use a library. **pint** is the common choice in scientific Python:

```python
import pint
ureg = pint.UnitRegistry()
Q_ = ureg.Quantity

force = 2.5 * ureg.kilonewton
area = (40 * ureg.mm) * (10 * ureg.mm)
stress = (force / area).to("MPa")
print(stress)                         # 6.25 megapascal

flow = 40 * ureg("L/min")
print(flow.to("m**3/s"))              # 0.000667 m³/s

T = Q_(25, ureg.degC).to("K")         # 298.15 kelvin (handles the offset)

(3 * ureg.m) + (2 * ureg.s)           # raises DimensionalityError
```

pint works with NumPy arrays (`np.array([1, 2, 3]) * ureg.m`), and `.magnitude` / `.m_as("unit")` give you the plain number when a function needs one. Other options are `astropy.units` (astronomy and physics) and `unyt`. pint is not loaded in this course's browser exercises, so the Do tab builds the dimension-checking idea yourself; install pint with `pip install pint` for local work.

## Significant figures and reporting

Units come with precision. Report results with a sensible number of significant figures and the unit, `6.25 MPa`, not `6.249999999 MPa` or a bare `6.25`. In tables, put the unit in the column header (`Stress (MPa)`).

## Common mistakes

- **Mixing mm and m** in one formula, especially in areas and volumes, where the error is a factor of 10⁶ or 10⁹.
- **°C in an absolute-temperature formula** (ideal gas, Arrhenius, Stefan-Boltzmann).
- **Gauge versus absolute pressure** (psig versus psia, or kPa(g) versus kPa(a)).
- **Degrees versus radians**: `np.sin(30)` is the sine of 30 radians.
- **Pound-mass versus pound-force**, and **kWh versus kW**.
- **Converting halfway through** a calculation instead of at the edges.

## Quick recap

- Convert inputs to SI on entry, compute in SI, convert on output; name conversion constants.
- Only add or compare quantities with equal dimensions; `exp`, `log`, `sin` need dimensionless arguments.
- Use pint (or astropy.units) to make the computer enforce this; watch temperature offsets.
- Report results with units and sensible precision.

## Try it

In the **Do** tab you will finish a small `Quantity` class that tracks dimensions, then use it to compute a speed and a force and catch an illegal addition.


Carry units through a calculation by converting to SI at the edges, checking dimensions, and using a units library such as pint.

Units-aware calculations

The Mars Metric Mistake

Units in Python

## Why this matters

Instruments rarely give you a tidy CSV. A thermocouple logger, a UV-Vis spectrometer, a tensile tester or a plate reader exports a file with a block of metadata at the top, column names that include units, a semicolon delimiter because the software was configured in Europe, decimal commas, and markers such as `---`, `OVER` or `NaN` where a reading failed. Opening it in a spreadsheet and fixing it by hand works once, but it is slow, error-prone and leaves no record. A short, documented Python function turns every future export into clean data the same way.

## A typical export

```text
Instrument: TC-08 Thermocouple Logger
Serial: A1234
Operator: J. Rivera
Exported: 2026-03-14 09:30
[Data]
Time (s);Temperature (°C);Signal (mV)
0;21,4;0,812
10;21,9;0,830
20;---;0,851
30;23,1;0,866
40;23,8;OVER
...
```

Before writing code, **open the raw file in a text editor** (not a spreadsheet, which silently reformats it) and note:

1. How many lines of metadata come before the header, or what marker ends them (`[Data]` here).
2. The delimiter (`;`), the decimal separator (`,`) and the encoding (usually UTF-8; older Windows software may use `latin-1`).
3. What missing or out-of-range values look like (`---`, `OVER`).
4. The units in each column header.

## Reading it with pandas

`pd.read_csv` handles all of this with arguments:

```python
import io
import pandas as pd

def read_export(text):
    lines = text.splitlines()
    start = lines.index("[Data]") + 1           # data header is the next line
    df = pd.read_csv(
        io.StringIO("\n".join(lines[start:])),
        sep=";",
        decimal=",",
        na_values=["---", "OVER"],
    )
    df.columns = ["time_s", "temp_C", "signal_mV"]
    return df
```

With a real file you pass the path instead of `io.StringIO`, plus `encoding="utf-8"` and `skiprows=` or `comment="#"` as needed. Two useful arguments for messy files: `skipinitialspace=True` (strips spaces after delimiters) and `usecols=` (keeps only the columns you need).

**Rename columns** to short, code-friendly names with the unit in the name (`temp_C`, not `Temperature (°C)`). It makes later code readable and keeps the unit attached.

## Keep the metadata

The header lines are data too: serial number, operator, calibration date, sample ID. Parse them into a dictionary and keep them with the results:

```python
def read_metadata(text):
    meta = {}
    for line in text.splitlines():
        if line == "[Data]":
            break
        key, _, value = line.partition(":")
        meta[key.strip()] = value.strip()
    return meta
```

`str.partition(":")` splits on the **first** colon only, so the time `09:30` in the value survives.

## Check before you clean

```python
df = read_export(text)
print(df.dtypes)            # every column should be float64 or int64
print(df.isna().sum())      # missing values per column
print(df.describe())        # ranges: is anything physically impossible?
```

If a column you expect to be numeric shows `object`, a marker you did not list in `na_values` (or a stray unit, or a thousands separator) is hiding in it. Find it with `df[pd.to_numeric(df["temp_C"], errors="coerce").isna()]`.

## Cleaning, transparently

Decide what to do with bad rows and **record the decision**:

```python
n_before = len(df)
clean = df.dropna().reset_index(drop=True)
print(f"dropped {n_before - len(clean)} of {n_before} rows with missing values")
```

Options, from most to least conservative:

- **Drop** rows with any missing value (`dropna()`), as here.
- **Drop only where a key column is missing** (`dropna(subset=["temp_C"])`).
- **Interpolate** short gaps in smooth time series (`df["temp_C"].interpolate()`), and say so in the methods.
- **Flag** suspect values in a new Boolean column rather than deleting them.

Never overwrite the raw export. Keep the raw file read-only, and let your script produce a separate cleaned file (`clean.to_csv("run42_clean.csv", index=False)`), so anyone can rerun the cleaning from scratch.

## Large and binary formats

Bigger instruments use other formats: HDF5 (`h5py`, `pd.read_hdf`), NetCDF (`xarray`), Excel (`pd.read_excel`), vendor binaries with Python readers, or fixed-width text (`pd.read_fwf`). The habits are the same: inspect the raw file, parse with explicit options, check dtypes and ranges, clean transparently.

## Common mistakes

- **Opening and re-saving in Excel**, which can change decimal separators, dates and leading zeros.
- **Letting pandas guess** the delimiter or decimal separator: `21,4` silently becomes text or splits into two columns.
- **Unlisted missing markers** that turn a numeric column into `object` dtype.
- **Dropping rows without counting them**, so nobody knows how much data was lost.
- **Hard-coding a row number** for the header when a marker line (`[Data]`) is more robust to metadata changes.

## Quick recap

- Inspect the raw file first: metadata, delimiter, decimal mark, missing markers, units.
- `pd.read_csv(..., sep=";", decimal=",", na_values=[...])`; rename columns to `name_unit`.
- Parse and keep the metadata; check `dtypes`, `isna().sum()` and `describe()`.
- Clean with recorded, reversible decisions; never edit the raw file.

## Try it

In the **Do** tab you will parse the logger export above into a DataFrame, extract its metadata and clean out the failed readings.


Parse a real instrument export with metadata headers, odd delimiters, decimal commas and missing-value markers into a clean pandas DataFrame.

Reading instrument data

Python pandas read_csv // Load a CSV into pandas from a file or URL // index_col, na_values, nrows

Python Pandas Tutorial (Part 9): Cleaning Data - Casting Datatypes and Handling Missing Values

Open a free notebook environment: JupyterLite (jupyter.org/try-jupyter, no account needed) or Google Colab (colab.research.google.com, free with a Google account). Create a new Python notebook named `reproducible_demo`.

Add a markdown title cell with the notebook's purpose, your name, today's date and the data source (for this demo: "simulated spring measurements").

Add one setup cell with all imports (numpy, pandas), `SEED = 2026`, `rng = np.random.default_rng(SEED)` and any constants with units in comments.

Add a cell that simulates data: 10 loads from 1 to 10 N and extensions `0.8 * load + rng.normal(0, 0.1, 10)` mm. Put it in a pandas DataFrame with columns `load_N` and `extension_mm`.

Add an analysis cell that fits a straight line with `np.polyfit` and prints the spring compliance in mm/N, then a markdown cell explaining the result in one sentence.

Add a final cell that prints the Python, numpy and pandas versions.

Run Restart and Run All (JupyterLab: Kernel menu; Colab: Runtime menu). Check that the execution counters run 1, 2, 3... with no errors, and write down the fitted slope.

Run Restart and Run All a second time and confirm the slope is identical. Then change SEED, rerun, and note that it changes; change it back.

Download the notebook (.ipynb) and keep it in a project folder with a short `requirements.txt` listing the package versions you printed.

## Why this matters

Jupyter notebooks are the everyday tool of scientific Python: code, results, plots and explanation in one document. They are also notorious for being impossible to rerun. Cells were executed out of order, a variable was defined in a cell that was later deleted, a file path points to someone's desktop, the random seed was never set, and the package versions are unknown. Six months later, not even the author can reproduce Figure 3.

In engineering and science that is not a style issue. A supervisor checking a design calculation, a reviewer checking a paper, a regulator auditing a QC report or a colleague taking over your project all need to rerun your analysis and get the same answer. **Reproducibility** means: same inputs + same code + same environment = same outputs, every time.

## The one rule: Restart and Run All

Before you share, commit or report from a notebook, choose **Kernel > Restart Kernel and Run All Cells** (JupyterLab) or **Runtime > Restart and run all** (Google Colab). If it finishes without errors and the results match what you reported, the notebook is at least reproducible on your machine. If it fails, it was relying on hidden state.

The execution counters (`[1]`, `[2]`, ...) next to each cell are a quick check: in a notebook ready to share they run 1, 2, 3 from top to bottom with no gaps or jumps.

## A structure that works

```text
1. Title and purpose      What question does this answer? Author, date, data version.
2. Setup                  All imports, the random seed, constants, file paths, settings.
3. Load data              Read raw files (never modified) and print their shape and head.
4. Clean                  Each cleaning step with a short comment and a row count.
5. Analyse                Calculations, fits, simulations.
6. Results                Final figures and tables, saved to an outputs/ folder.
7. Environment            Package versions printed at the end.
```

Short markdown cells between sections explain the **why**, not just the what: "Rows with OVER are detector saturation; we drop them (2 of 480 rows)."

### The setup cell

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pathlib import Path

SEED = 2026
rng = np.random.default_rng(SEED)

DATA_DIR = Path("data")          # relative to the project folder
OUT_DIR = Path("outputs")
OUT_DIR.mkdir(exist_ok=True)

G = 9.81                         # m/s^2
```

Everything that might change between runs (seed, paths, constants, thresholds) lives here, in one place, with units.

## Habits that make notebooks rerunnable

1. **Relative paths only.** `data/run42.csv`, not `C:\Users\me\Desktop\run42.csv`. Keep data, notebook and outputs together in one project folder.
2. **Never edit raw data.** Load it, transform it in code, and save results separately.
3. **Set every seed** (NumPy's `default_rng`, plus any library-specific seeds).
4. **Put reusable code in functions**, and move long functions into a `.py` module you import. Notebooks are for the narrative; modules are for tested, reusable logic.
5. **No manual steps.** If you had to click something or copy a number by hand, the notebook is not reproducible. Write it in code.
6. **Record the environment.** Pin versions in a `requirements.txt` (for example `numpy==2.1.3`) or an `environment.yml`, and print versions at the end:

```python
import sys, numpy, pandas, matplotlib
print(sys.version)
print("numpy", numpy.__version__, "| pandas", pandas.__version__,
      "| matplotlib", matplotlib.__version__)
```

7. **Version control.** Keep the project in git. Notebooks store outputs inside the file, which makes diffs noisy; clearing outputs before committing (or using a tool such as `jupytext` or `nbstripout`) keeps history readable.
8. **Save figures from code** (`fig.savefig(OUT_DIR / "fig3_stress.pdf")`) rather than copying them from the screen.

## Sharing and running elsewhere

- **Google Colab** runs notebooks in the browser with numpy, pandas, matplotlib and SciPy preinstalled; a Google account is needed. Free tier limits apply, and files uploaded to a session disappear when it ends.
- **JupyterLite** runs a notebook entirely in your browser with no install and no account (it uses Pyodide, like this site's exercises).
- **Local Jupyter** via `pip install jupyterlab` or the Anaconda distribution.
- **Binder** can launch a public git repository as a live notebook, using your `requirements.txt`.

Whichever you use, test the notebook in a **fresh** environment at least once: a new Colab session or a new virtual environment. That is the closest you can get to being the next person who runs it.

## Common mistakes

- **Out-of-order execution**: results depend on a cell that was run, edited and never rerun.
- **Deleted cells** whose variables still live in memory.
- **Absolute paths** and files that exist only on your machine.
- **Unseeded randomness** in simulations, bootstraps or train/test splits.
- **Giant notebooks** with hundreds of cells and copy-pasted code instead of functions.
- **Unknown versions**: a function's default changed between library releases, and the numbers moved.

## Quick recap

- Restart and Run All before you share; execution counters should run 1, 2, 3...
- Structure: purpose, setup (imports, seed, paths, constants), load, clean, analyse, results, environment.
- Relative paths, untouched raw data, seeds, functions, no manual steps, pinned versions, git.
- Test in a fresh environment.

## Try it

The **Do** tab walks you through building a small notebook in Colab or JupyterLite that passes the Restart and Run All test.


Structure a Jupyter notebook so that someone else can rerun it from top to bottom and get the same results.

Reproducible notebooks

Reproducible Data Analysis in Jupyter, Part 1/10: Loading and Visualizing Data

Reproducible Data Analysis in Jupyter, Part 10/10: Cleaning-up the Notebook

Carry units through calculations, parse and clean instrument exports with pandas, and build notebooks that someone else can rerun and trust.