Load, clean, reshape and explore real data with Python and pandas, then turn the results into clear findings.

Intro to Data Analysis with Python

## Why it matters

Almost every analysis starts the same way: someone hands you a file, usually a CSV, and asks a question about it. Before you can answer anything you need to get that file into a **DataFrame** and check what you actually have. Skipping that check is how analysts end up averaging a column that was secretly text.

## The DataFrame in one minute

A pandas `DataFrame` is a table: rows, named columns, and an **index** (row labels, by default 0, 1, 2, ...). Each column is a `Series`, a one-dimensional array with a single data type (`dtype`), such as `int64`, `float64`, `bool`, `datetime64` or text.

```python
import pandas as pd

df = pd.DataFrame({
    "sample": ["S1", "S2", "S3"],
    "temp_c": [21.5, 22.1, 20.8],
})
df["temp_c"]          # a Series
df["temp_c"].mean()   # 21.466...
```

## Loading a CSV

`pd.read_csv` reads a file path, a URL or any file-like object:

```python
df = pd.read_csv("readings.csv")
df = pd.read_csv("https://example.org/data/readings.csv")
```

In the browser exercises there is no file system, so the data lives in a string and `io.StringIO` makes it look like a file:

```python
import io

RAW = """station,date,temp_c,humidity
A,2026-03-01,4.5,81
B,2026-03-01,6.1,77
A,2026-03-02,5.2,79
"""
df = pd.read_csv(io.StringIO(RAW))
```

Options you will use often:

| Option | Use it when |
|---|---|
| `sep=";"` | the file uses semicolons or tabs (`sep="\t"`) |
| `usecols=["a", "b"]` | you only need some columns |
| `parse_dates=["date"]` | a column holds dates |
| `na_values=["-", "ERR"]` | the file marks missing values with custom text (common markers such as empty cells, `NA`, `n/a` and `NaN` are recognised by default) |
| `dtype={"zip": str}` | codes with leading zeros must stay text |
| `nrows=1000` | you want a quick look at a huge file |
| `encoding="latin-1"` | you get a `UnicodeDecodeError` |

## The first five checks

Run these every time you load data:

```python
df.shape        # (rows, columns), e.g. (3, 4)
df.head()       # first 5 rows (df.tail() for the last)
df.columns      # the column names
df.dtypes       # the type of each column
df.info()       # all of the above plus non-null counts
```

Then summarise the numbers:

```python
df.describe()                  # count, mean, std, min, quartiles, max
df["station"].value_counts()   # how often each value appears
df["station"].nunique()        # number of distinct values
```

Each answer is a question about your data. Is the row count what you expected? Are the numeric columns really numeric? Does `count` in `describe()` show missing values? Does `value_counts()` reveal `"A"` and `"a "` as different stations?

## Selecting numeric columns

`select_dtypes` filters columns by type, which is handy before summarising:

```python
numeric = df.select_dtypes(include="number")
list(numeric.columns)   # ['temp_c', 'humidity']
```

## Reading dtypes

If a column you expected to be numeric shows as `object` or `str`, at least one value in it is not a clean number: a stray `"ERR"` or `"-"`, a comma used as a decimal point, a unit such as `"5 kg"`. pandas then keeps the whole column as text, and `.mean()` will fail or give nonsense. You will fix this in the next lesson; for now, notice it.

```python
bad = pd.read_csv(io.StringIO("x\n1\n2\n-\n"))
bad.dtypes                   # x is text, because of "-"
fixed = pd.read_csv(io.StringIO("x\n1\n2\n-\n"), na_values=["-"])
fixed.dtypes                 # x float64, with NaN for the missing value
```

## A worked example

```python
df = pd.read_csv(io.StringIO(RAW), parse_dates=["date"])
print(df.shape)                       # (3, 4)
print(df.dtypes)                      # station text, date datetime64, temp_c float64, humidity int64
print(df["temp_c"].mean().round(2))   # 5.27
print(df["station"].value_counts())   # A 2, B 1
```

In four lines you know the size, the types, a headline number and the balance of stations.

## Common mistakes

- **Not looking at the data.** Always run `head()` and `info()` before computing anything.
- **Lost leading zeros.** ZIP codes and IDs like `00742` become `742` unless you pass `dtype={"zip": str}`.
- **Header problems.** If the first row is data, pass `header=None` and set `names=[...]`. If there are notes above the header, use `skiprows`.
- **Calling `df.shape()`.** `shape` is an attribute, not a method: `df.shape`.
- **Assuming dates are dates.** Without `parse_dates` or `pd.to_datetime`, a date column is just text.

## Quick recap

- `pd.read_csv` loads a file, URL or `io.StringIO` string into a DataFrame.
- `shape`, `head`, `dtypes`, `info`, `describe` and `value_counts` are the first checks.
- A numeric column showing as text means at least one dirty value.
- `select_dtypes(include="number")` picks the numeric columns.

## Try it

In the Do tab you will load a small weather-station CSV and record its shape, column names, numeric columns and a mean temperature.


Load a CSV into a pandas DataFrame and inspect its shape, columns, types and values before analysing it.

Loading a CSV

Python Pandas Tutorial (Part 1): Getting Started with Data Analysis - Installation and Loading Data

Python Pandas Tutorial (Part 2): DataFrame and Series Basics - Selecting Rows and Columns

## Why it matters

Real data is messy. Survey answers have stray spaces, sensor logs contain `"ERR"`, exports duplicate rows, and somebody typed `"unknown"` into a number column. Cleaning is often most of the work in an analysis, and mistakes here quietly change every result after it. The goal is a clean table produced by **code you can rerun**, never by hand edits.

## Find the problems first

```python
df.info()                  # dtypes and non-null counts
df.isna().sum()            # missing values per column
df.duplicated().sum()      # fully duplicated rows
df["city"].value_counts()  # spelling and spacing variants
```

Suppose the data looks like this:

```text
   city      temp_c
0  Boston    21.5
1  boston    22.0
2  Denver    ERR
3  Boston    21.5
4  Austin     30.25
```

Problems: `"boston"` and `" Boston"` differ in case and spacing, `temp_c` is text because of `"ERR"` (pandas already reads empty cells, `NA` and `n/a` as missing, but not arbitrary codes), and row 3 may duplicate row 0 once the city is cleaned.

## Fixing text: the .str accessor

String methods on a whole column live under `.str`:

```python
df["city"] = df["city"].str.strip().str.title()
# " boston " -> "Boston"
```

Other useful ones: `.str.lower()`, `.str.replace("$", "", regex=False)`, `.str.contains("lab")`, `.str.split("-")`.

## Fixing types

`pd.to_numeric` converts text to numbers. With `errors="coerce"`, anything that cannot be converted becomes `NaN` (missing) instead of raising an error:

```python
df["temp_c"] = pd.to_numeric(df["temp_c"], errors="coerce")
```

Strip units or symbols first if needed:

```python
df["price"] = pd.to_numeric(df["price"].str.replace("$", "", regex=False), errors="coerce")
```

For dates, use `pd.to_datetime(df["date"], errors="coerce")`. Use `.astype(int)` only when you are sure there are no missing values.

## Handling missing values

`NaN` means "no value". Most pandas statistics skip it (`mean()` ignores NaN), but you still have to decide what missing **means** in your data.

```python
df.dropna(subset=["temp_c"])        # drop rows missing temp_c
df["temp_c"].fillna(0)              # fill with a value (rarely right for measurements)
df["temp_c"].fillna(df["temp_c"].median())   # fill with the median
```

Rules of thumb:

- **Drop** when the row cannot be used without that value and there are few such rows.
- **Fill** when a sensible value exists, such as 0 for "no sales recorded", and say so in your notes.
- **Keep as NaN** when later steps can skip it and dropping would throw away useful columns.
- **Count before and after.** Report how many rows you removed and why.

## Removing duplicates

```python
df = df.drop_duplicates()                     # rows identical in every column
df = df.drop_duplicates(subset=["order_id"])  # one row per order_id (keeps the first)
```

Clean text **before** removing duplicates, otherwise `"Boston"` and `"boston "` look different and survive.

## Putting it in a function

Wrap the steps in a function that returns a new DataFrame, so the raw data stays untouched and you can rerun the cleaning:

```python
def clean_temps(raw):
    df = raw.copy()
    df["city"] = df["city"].str.strip().str.title()
    df["temp_c"] = pd.to_numeric(df["temp_c"], errors="coerce")
    df = df.dropna(subset=["temp_c"])
    df = df.drop_duplicates()
    return df.reset_index(drop=True)
```

`reset_index(drop=True)` renumbers rows 0, 1, 2, ... after rows were removed.

## Common mistakes

- **Editing the original in place without meaning to.** Start with `df = raw.copy()`.
- **Assigning to a filtered slice.** `df[df["x"] > 0]["y"] = 1` may not change `df`. Use `df.loc[df["x"] > 0, "y"] = 1`.
- **Filling measurements with 0.** A missing temperature is not 0 degrees.
- **Deduplicating before normalising text.** Near-duplicates survive.
- **Silently dropping rows.** Always know how many rows you lost.

## Quick recap

- Inspect with `info`, `isna().sum()`, `duplicated().sum()` and `value_counts()`.
- `.str.strip().str.title()` normalises text; `pd.to_numeric(..., errors="coerce")` fixes numbers.
- Decide per column whether to drop, fill or keep missing values, and record the choice.
- Normalise text first, then `drop_duplicates`, inside a function that returns a copy.

## Try it

In the Do tab you will write `clean(df)` for a messy city-temperature table: fix the city names, convert temperatures, drop unusable rows and remove duplicates.


Fix text, types, missing values and duplicates with a rerunnable pandas cleaning function.

Cleaning messy data

Python Pandas Tutorial (Part 9): Cleaning Data - Casting Datatypes and Handling Missing Values

Data Cleaning in Pandas | Python Pandas Tutorials

## Why it matters

A good chart answers one question so clearly that the reader gets it in five seconds. A bad chart shows everything and answers nothing. Analysts are judged by whether people understand and act on their work, so learning to make **one** clear chart is worth more than learning twenty chart types.

## Start from the question, not the chart

Write the question first, in plain words:

- "Which region sold the most units this quarter?"
- "Did average response time improve after the change in June?"
- "Is reading B consistently higher than reading A?"

The question decides the chart:

| Question type | Chart |
|---|---|
| Compare categories | bar chart (horizontal if labels are long), sorted |
| Change over time | line chart |
| Relationship between two numbers | scatter plot |
| Distribution of one number | histogram or box plot |
| Part of a whole (2 to 4 parts) | stacked bar; pie only if parts are few and very different |

## Prepare the data first

Most of the work is computing the exact numbers you will plot. For "Which region sold the most units?":

```python
by_region = (
    df.groupby("region")["units"].sum()
      .sort_values(ascending=False)
)
# North 40, East 31, West 18, South 11
top = by_region.index[0]
share = by_region.iloc[0] / by_region.sum() * 100   # 40.0
```

Sorting matters: a sorted bar chart lets the reader see the ranking without reading numbers.

## Draw it with matplotlib

matplotlib is the standard Python plotting library, and pandas uses it for `df.plot()`. In a notebook:

```python
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(6, 3.5))
by_region.sort_values().plot.barh(ax=ax, color="#9ca3af")
ax.barh(top, by_region[top], color="#2563eb")      # highlight the answer
ax.set_title(f"{top} sold the most units in Q3 ({share:.0f}% of total)")
ax.set_xlabel("Units sold")
ax.set_ylabel("")
ax.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
fig.savefig("units_by_region.png", dpi=150)
```

Note the choices:

- **The title states the finding**, not just the topic. "North sold the most units in Q3 (40% of total)" beats "Units by region".
- **One highlight colour** for the answer; everything else is grey.
- **Axis labels with units**; remove labels that add nothing.
- **No clutter**: no 3D, no shadows, fewer gridlines and borders.
- **Bars start at zero.** Truncated bar axes exaggerate differences.

(The browser exercises cannot draw charts, so the Do tab focuses on preparing the numbers and the title. Run the plotting code in a notebook from the previous module.)

## Check the chart tells the truth

- Does the y-axis of a bar chart start at 0?
- Are categories sorted in a meaningful order (by value, or naturally, like months)?
- Would a colour-blind reader see the highlight? Blue against grey is safe; red against green is not.
- Is the time period and data source written somewhere (caption or subtitle)?
- Have you avoided dual y-axes, which can make any two lines look related?

## Common mistakes

- **Chart first, question later.** You end up with a pretty picture that says nothing.
- **Unsorted bars.** Alphabetical order hides the ranking.
- **Too many series.** More than about five lines on one chart becomes spaghetti; highlight one or split into small multiples.
- **Pie charts with many slices.** People compare lengths far better than angles.
- **A title that only names the variables.** Say what the reader should notice.

## Quick recap

- Write the question, then pick the chart type that answers it.
- Compute and sort the exact numbers first; that is most of the work.
- Title = the finding. Highlight the answer, grey the rest, label axes with units.
- Bars start at zero; keep it simple.

## Try it

In the Do tab you will prepare the data for a "units by region" chart: the sorted totals, the top region, its share and a finding-style title.


Pick the chart that answers one question, prepare its numbers in pandas and give it a finding-style title.

Tell a story with one chart

Matplotlib Tutorial (Part 2): Bar Charts and Analyzing Data from CSVs

Telling Stories with Data in 3 Steps (Quick Study)

## Why it matters

Most questions are about a **subset**: orders from one region, samples above a threshold, patients in a date range, the ten largest accounts. Selecting the right rows and columns, and ordering them, is the most common thing you will do in pandas. It is also where many silent bugs hide, so it is worth learning the clear, explicit way.

We will use this table:

```python
orders = pd.DataFrame({
    "order_id": [101, 102, 103, 104, 105, 106],
    "region":   ["East", "West", "East", "North", "East", "West"],
    "product":  ["Kettle", "Toaster", "Blender", "Kettle", "Toaster", "Blender"],
    "units":    [12, 3, 15, 7, 4, 10],
    "price":    [25.0, 40.0, 60.0, 25.0, 40.0, 60.0],
})
orders["revenue"] = orders["units"] * orders["price"]
```

## Selecting columns

```python
orders["units"]                    # one column -> Series
orders[["order_id", "units"]]      # a list of columns -> DataFrame
```

Note the double brackets: the inner brackets are a Python list of names.

## Boolean masks

A comparison on a column gives a **mask**: a Series of `True`/`False`, one per row.

```python
mask = orders["units"] >= 10
# [True, False, True, False, False, True]
orders[mask]          # only the True rows: 101, 103, 106
```

Combine conditions with `&` (and), `|` (or) and `~` (not), and **wrap each condition in parentheses**:

```python
east_big = orders[(orders["region"] == "East") & (orders["units"] >= 10)]
# orders 101 and 103

not_east = orders[~(orders["region"] == "East")]
```

Python's `and`/`or` do not work here: they try to treat a whole Series as one `True` or `False` and raise `ValueError: The truth value of a Series is ambiguous`.

For "one of several values" use `isin`; for ranges use `between` (inclusive at both ends):

```python
orders[orders["product"].isin(["Kettle", "Toaster"])]
orders[orders["price"].between(30, 60)]
```

## .loc and .iloc

`.loc` selects by **label** (index labels and column names), and accepts masks. It is the clearest way to pick rows and columns at once:

```python
orders.loc[orders["units"] < 5, ["order_id", "product"]]
#    order_id  product
# 1       102  Toaster
# 4       105  Toaster
```

`.iloc` selects by **position**, like list slicing:

```python
orders.iloc[0]        # first row
orders.iloc[:3, :2]   # first 3 rows, first 2 columns
```

Rule of thumb: use `.loc` with conditions and names in analysis code; use `.iloc` when position really is what you mean ("the first row after sorting"). Note that `.loc` slices include the end label, while `.iloc` slices exclude the end position.

`query` is a readable alternative for simple filters: `orders.query("region == 'East' and units >= 10")`.

## Sorting

```python
orders.sort_values("revenue", ascending=False)              # largest first
orders.sort_values(["region", "units"], ascending=[True, False])  # region A-Z, then units high-low
orders.nlargest(3, "revenue")                                # the top 3 rows by revenue
```

After filtering and sorting the index keeps the old labels (for example 2, 0, 5). Use `.reset_index(drop=True)` when you want 0, 1, 2 again, and remember that `.iloc[0]` means "first row now" while `.loc[0]` means "the row whose label is 0".

## A worked example

"List the East orders worth at least 300, largest first, showing only id and revenue":

```python
result = (
    orders.loc[(orders["region"] == "East") & (orders["revenue"] >= 300), ["order_id", "revenue"]]
          .sort_values("revenue", ascending=False)
          .reset_index(drop=True)
)
#    order_id  revenue
# 0       103    900.0
# 1       101    300.0
```

Wrapping a chain in parentheses lets you put each step on its own line.

## Common mistakes

- **Using `and`/`or` instead of `&`/`|`.** Raises the "truth value is ambiguous" error.
- **Missing parentheses.** `df["a"] == 1 & df["b"] == 2` is parsed wrongly because `&` binds tighter than `==`.
- **Forgetting that sorting returns a new DataFrame.** `orders.sort_values("units")` on its own line changes nothing; assign the result.
- **Mixing up `.loc[0]` and `.iloc[0]`** after filtering or sorting.
- **Filtering, then assigning into the result** and expecting the original to change. Use `.loc[mask, "col"] = value` on the original.

## Quick recap

- `df[mask]` keeps rows where the mask is True; combine with `&`, `|`, `~` and parentheses.
- `isin` for sets of values, `between` for ranges.
- `.loc[rows, cols]` by label or mask; `.iloc` by position.
- `sort_values` (assign the result), `nlargest` for top N, `reset_index(drop=True)` to renumber.

## Try it

In the Do tab you will filter, select and sort an orders table: big East orders, a two-column view of small orders, and the top three orders by revenue.


Select rows and columns with boolean masks, .loc, .iloc and isin, and order results with sort_values.

Filtering and sorting

Python Pandas Tutorial (Part 4): Filtering - Using Conditionals to Filter Rows and Columns

Python Pandas Tutorial (Part 7): Sorting Data

Load a CSV into a DataFrame, clean types, missing values and duplicates, select and sort rows, and prepare the numbers for one clear chart.