Load, clean, reshape and explore real data with Python and pandas, then turn the results into clear findings.

Intro to Data Analysis with Python

## Why it matters

Most people meet data in a spreadsheet, and spreadsheets are good tools. You can see every cell, sort with a click and build a chart in seconds. So why does nearly every data analyst, data scientist, statistician and lab scientist job listing ask for Python as well?

The short answer: once an analysis has to be **repeated, checked or scaled**, a script beats a sequence of clicks. This lesson compares the two honestly, so you know when to reach for each.

## What a spreadsheet does well

- **Seeing the data.** Scrolling through rows is the fastest way to spot an obviously wrong value.
- **Small, one-off questions.** "What was the total in March?" is a `SUM` and you are done.
- **Sharing with non-programmers.** Almost everyone can open an `.xlsx` file.
- **Light what-if modelling.** Change an input cell and watch the result update.

If the job is a single, small, look-once question, a spreadsheet is often the right answer. Do not let anyone tell you otherwise.

## Where Python pulls ahead

**1. Repeatability.** Suppose you clean a monthly export: trim spaces, fix dates, remove test rows, then build a summary. In a spreadsheet those are a dozen manual steps you must redo, in the same order, every month. In Python they are a script:

```python
import pandas as pd

df = pd.read_csv("orders_2026_10.csv")
df["region"] = df["region"].str.strip().str.title()
df = df[df["customer"] != "TEST"]
summary = df.groupby("region")["revenue"].sum()
print(summary)
```

Next month you change one file name and run it again. The steps cannot be done out of order or skipped by accident.

**2. An audit trail.** The script *is* the record of what you did. A colleague (or you, six months later) can read every step. A spreadsheet hides its logic in cell formulas and in actions nobody wrote down, such as "I deleted the duplicate rows by hand".

**3. Size.** Excel stops at 1,048,576 rows, and it gets slow well before that. pandas handles millions of rows on a laptop, and other Python tools go further.

**4. Combining sources.** Joining a CSV export, a database table and a JSON API response is a few lines of Python. In a spreadsheet it usually means copy, paste and `VLOOKUP`, which breaks silently when a key is missing.

**5. The wider toolkit.** The same language gives you statistics (`scipy`, `statsmodels`), charts (`matplotlib`, `seaborn`), machine learning (`scikit-learn`) and automation (scheduled reports, emails, file handling). Learning Python for analysis is also the first step toward those.

## The libraries you will use

| Library | What it gives you |
|---|---|
| **pandas** | The `DataFrame`: a table with named columns, plus tools to load, clean, filter, group and join |
| **NumPy** | Fast arrays and maths; pandas is built on top of it |
| **matplotlib** | Charts; pandas uses it for `df.plot()` |
| **Jupyter** | Notebooks that mix code, output, charts and notes in one document |

This course uses pandas and NumPy in the browser exercises. Charts and notebooks are covered in the readings and the notebook warm-up.

## A worked comparison

Task: from a 50,000-row sales export, find the average order value per region, excluding refunds, every Monday.

**Spreadsheet route:** open the file, filter out refunds, insert a pivot table, set rows = region and values = average of order value, copy the result into the weekly report. About 5 minutes, every week, and easy to get subtly wrong (forgetting the refund filter one week, or including the totals row).

**Python route:** about 15 minutes to write the first time, then seconds each week:

```python
df = pd.read_csv("sales.csv")
clean = df[df["type"] != "refund"]
print(clean.groupby("region")["order_value"].mean().round(2))
```

By the third week the script has paid for itself, and it never forgets the filter.

## Working with both

In real jobs the two tools live side by side. A common pattern is to explore a sample in a spreadsheet, write the repeatable pipeline in Python, and export the final summary back to Excel for the people who need it (`summary.to_excel("report.xlsx")`). Microsoft now even offers Python inside Excel. Choose the tool that fits the job, not the one that feels most impressive.

## Common mistakes

- **Thinking Python replaces understanding the data.** You still need to look at rows, question odd values and know what each column means.
- **Rewriting a five-second spreadsheet task as a script.** Automate what you will repeat or what must be auditable.
- **Editing the raw file by hand, then scripting the rest.** The manual edit is invisible to the script. Keep the raw data untouched and put every change in code.
- **Treating a spreadsheet's display as the value.** A cell showing `3.1` may hold `3.14159`. Python shows you the real value, which sometimes surprises people.

## Quick recap

- Spreadsheets are great for looking, quick one-off questions and sharing.
- Python wins when work must be repeated, audited, scaled or combined from several sources.
- pandas (tables), NumPy (arrays), matplotlib (charts) and Jupyter (notebooks) are the core toolkit.
- Most analysts use both: explore, script, then export.

## Try it

This lesson's Do tab is a short quiz on choosing the right tool.


Compare Python with spreadsheets and learn when each is the right tool for an analysis.

Why Python for data?

Python for Data Analysis: Getting Started

Python vs. Excel for 7 Data Analysis Tasks

## Why it matters

Before you reach for pandas, it pays to be fluent with plain Python **lists** and **dictionaries**. Data arrives as JSON from APIs, as rows from the `csv` module, or as small lookup tables you build yourself. All of these are lists of dictionaries. Knowing how to reshape them quickly, with **comprehensions**, makes you faster and helps you understand what pandas does for you later.

## Records: a list of dictionaries

A common shape for tabular data in plain Python is one dictionary per row:

```python
orders = [
    {"id": 1, "region": "East", "product": "Kettle", "units": 3, "price": 25.0},
    {"id": 2, "region": "West", "product": "Toaster", "units": 1, "price": 40.0},
    {"id": 3, "region": "East", "product": "Toaster", "units": 2, "price": 40.0},
    {"id": 4, "region": "North", "product": "Kettle", "units": 5, "price": 25.0},
]
```

Each dictionary is a **record**, and the keys play the role of column names. You read a field with square brackets: `orders[0]["region"]` is `"East"`. Use `.get(key, default)` when a field might be missing: `row.get("discount", 0)`.

## List comprehensions

A list comprehension builds a new list from an existing one in a single expression:

```python
revenues = [o["units"] * o["price"] for o in orders]
# [75.0, 40.0, 80.0, 125.0]
```

Read it left to right as "the value `units * price`, for each `o` in `orders`". It is equivalent to:

```python
revenues = []
for o in orders:
    revenues.append(o["units"] * o["price"])
```

Add an `if` at the end to **filter**:

```python
east_ids = [o["id"] for o in orders if o["region"] == "East"]
# [1, 3]
```

And use a conditional expression at the front to **transform** values differently:

```python
sizes = ["large" if o["units"] >= 3 else "small" for o in orders]
# ['large', 'small', 'small', 'large']
```

Note the position: a filter `if` goes at the end and has no `else`; a value choice `x if cond else y` goes at the front.

## Dictionary and set comprehensions

The same idea builds dictionaries, which is ideal for lookups:

```python
price_by_id = {o["id"]: o["price"] for o in orders}
# {1: 25.0, 2: 40.0, 3: 40.0, 4: 25.0}

price_by_id[3]   # 40.0, found instantly without scanning the list
```

A **set** comprehension gives the distinct values:

```python
regions = {o["region"] for o in orders}
# {'East', 'West', 'North'}  (sets have no fixed order)
sorted(regions)  # ['East', 'North', 'West']
```

## Grouping and totals

Comprehensions are great for one-to-one reshaping. For **totals by group**, a plain loop with a dictionary is clearer:

```python
units_by_region = {}
for o in orders:
    region = o["region"]
    units_by_region[region] = units_by_region.get(region, 0) + o["units"]

# {'East': 5, 'West': 1, 'North': 5}
```

The `.get(region, 0)` handles the first time a region is seen. The standard library's `collections.Counter` and `defaultdict` do the same with less code:

```python
from collections import defaultdict

totals = defaultdict(float)
for o in orders:
    totals[o["product"]] += o["units"] * o["price"]
# {'Kettle': 200.0, 'Toaster': 120.0}
```

This "group, then aggregate" pattern is exactly what pandas' `groupby` does, which you will meet in module 3.

## Sorting records

`sorted` takes a `key` function that says what to sort by:

```python
by_units = sorted(orders, key=lambda o: o["units"], reverse=True)
[o["id"] for o in by_units]   # [4, 1, 3, 2]
```

## Common mistakes

- **Mutating records you did not mean to change.** `o["units"] = 0` inside a loop changes the original data. Build new dictionaries instead: `{**o, "units": 0}`.
- **Huge, nested comprehensions.** If a comprehension needs two `for`s and three conditions, write a loop. Clarity beats cleverness.
- **KeyError on missing fields.** Real JSON often omits keys. Use `row.get("field")` and decide what a missing value means.
- **Relying on set order.** Sets are unordered; wrap them in `sorted()` before showing them.
- **Comparing floats with `==`.** `0.1 * 3 == 0.3` is `False`. Use `math.isclose` or round when checking money or measurements.

## Quick recap

- A list of dictionaries is plain Python's table: one dictionary per record.
- `[expr for x in items if cond]` transforms and filters in one line.
- `{key: value for x in items}` builds fast lookups; `{x for ...}` gives distinct values.
- For totals by group, loop with `dict.get` or use `defaultdict` / `Counter`.

## Try it

In the Do tab you will write three small functions over a list of order records: revenue per order, products from large orders, and total units per region.


Reshape lists of records with list, dictionary and set comprehensions, and total values by group.

Lists, dicts, and comprehensions

Python Tutorial: Comprehensions - How they work and why you should be using them

How to Use Python Dictionaries + Lists of Dicts

Open a new Python notebook and add a markdown cell at the top with a title and one sentence describing what the notebook will do.

In a code cell, `import pandas as pd` and build a small DataFrame by hand with 6 rows and 3 columns (for example sample, group, reading). Run it with Shift+Enter and check the table appears.

In new cells, run `df.head()`, `df.info()` and `df.describe()`, and write one markdown sentence about what each one told you.

Add a cell that computes the mean of the numeric column for each group with `df.groupby("group")["reading"].mean()`.

Create hidden state on purpose: add a cell `df = df[df["reading"] > df["reading"].min()]` and run it twice. Rerun the groupby cell and note how the result changed.

Restart the kernel and run all cells. Compare the final numbers with what you saw before the restart.

Add a final markdown "Findings" cell with two sentences, then save or download the notebook as an .ipynb file.

## Why it matters

Most real analysis happens in a **notebook**: a document that mixes runnable code, its output, tables, charts and written notes. Jupyter is the standard format (`.ipynb`), and Google Colab, JupyterLab, VS Code and Kaggle all open it. Being comfortable in a notebook, and knowing its traps, is a day-one skill for analysts and scientists.

The browser exercises in this course run Python for you, so you do not need to install anything. This lesson gets you running a real notebook as well, because that is where you will do your own projects.

## Three ways to get a notebook

1. **JupyterLite (no account, nothing to install).** Visit https://jupyter.org/try-jupyter and open a Python notebook. It runs in your browser using the same Pyodide engine as our exercises, and pandas is available.
2. **Google Colab (free Google account).** https://colab.research.google.com gives you a notebook on Google's servers. Files are saved to your Google Drive. The free tier is enough for this course; you never need to pay or add a card.
3. **Local Jupyter (install).** Install Python, then in a terminal run `python -m pip install jupyterlab pandas` and `jupyter lab`. This is what you will use for private or large data.

## Cells and the kernel

A notebook is a list of **cells**:

- **Code cells** run Python. The value of the last line is displayed automatically, which is why `df.head()` on its own shows a table.
- **Markdown cells** hold notes, headings and explanations.

Behind the notebook is a **kernel**: a running Python process that remembers every variable you have created. Press **Shift+Enter** to run a cell and move to the next one. The number in `[3]` next to a cell shows the order cells were run in.

```python
# Cell 1
import pandas as pd
data = {"sample": ["A", "B", "C"], "reading": [4.2, 3.9, 5.1]}
df = pd.DataFrame(data)
df
```

```python
# Cell 2
df["reading"].mean()
```

Cell 2 works only because Cell 1 already ran in this kernel.

## The big trap: hidden state

Because the kernel remembers everything, you can run cells out of order, edit a cell and rerun it, or delete a cell whose variable still exists in memory. The notebook can then show results that **no longer match the code on the page**.

Example: you create `df`, then filter it with `df = df[df["reading"] > 4]`, then run that cell twice while experimenting. Nothing looks wrong, but `df` has been filtered twice and a later average is different from what a fresh run would give.

The fix is a habit: **Restart the kernel and run all** (in Jupyter: Kernel > Restart Kernel and Run All Cells; in Colab: Runtime > Restart session and run all) before you trust or share a result. If it runs top to bottom without errors and gives the same numbers, your notebook is reproducible.

## A tidy notebook layout

A notebook others can follow usually looks like this:

1. A title and a one-paragraph purpose in markdown.
2. One cell with all the imports.
3. Loading the data (with the file name or URL written down).
4. Cleaning steps, each with a short markdown note on why.
5. Analysis and charts.
6. A short "Findings" markdown cell at the end.

Avoid giant cells. Small cells that each do one thing are easier to rerun and debug.

## Useful notebook features

- `df.head()`, `df.info()` and `df.describe()` give a quick look at any table.
- `?` after a name shows help: `pd.read_csv?`.
- Tab completion suggests column names and methods.
- `%timeit` measures how long a line takes (a "magic" command, notebook only).

## Common mistakes

- **Trusting a notebook that has never been run top to bottom.** Restart and run all first.
- **Variables defined in deleted cells.** The code is gone but the value is still in memory. A restart exposes this.
- **Printing a 100,000-row table.** Use `.head()` or `.sample(5)`.
- **Uploading private or personal data to a cloud notebook.** Check your organisation's rules first; use local Jupyter for sensitive data.

## Quick recap

- A notebook = code cells + markdown cells + a kernel that remembers state.
- Shift+Enter runs a cell; the last expression is displayed.
- Out-of-order runs create hidden state; restart and run all before trusting results.
- JupyterLite and Colab are free ways to start; local Jupyter for private data.

## Try it

The Do tab walks you through a short notebook session from a blank page to a restart-and-run-all check.


Run a Jupyter notebook end to end and learn to avoid hidden state with restart and run all.

Notebook warm-up

Jupyter Notebook Tutorial: Introduction, Setup, and Walkthrough

Google Colab for Python Beginners - Visually Explained

Why analysts use Python alongside spreadsheets, how to reshape records with lists, dictionaries and comprehensions, and how to run a notebook from top to bottom.