Exploratory analysis

Why it matters

Exploratory data analysis (EDA) is the structured first look at a dataset before you trust it. It answers: What is in here? What is broken? What is surprising? Skipping EDA is how analysts report averages inflated by a single typo, or "trends" caused by a month of missing data. A short, repeatable checklist catches most of these problems in minutes.

The EDA checklist

Work through these in order and write down what you find.

1. Shape and meaning. How many rows and columns? What is one row (one order, one sample, one day)? Does the count match what you were told?

df.shape
df.head()
df.sample(5, random_state=1)   # random rows, reproducible

2. Types. Are numbers numeric, dates datetime, IDs text?

df.dtypes

3. Missing values. How many per column, and are they random or concentrated (one site, one month)?

df.isna().sum()
df.isna().mean().round(3)      # share missing per column

4. Duplicates. Whole-row duplicates and duplicate keys.

df.duplicated().sum()
df["order_id"].duplicated().sum()

5. Ranges and summaries. Do minimums and maximums make sense? Negative ages, delivery times of 0 or 900 minutes, percentages over 100?

df.describe()

6. Categories. Spelling variants, unexpected codes, rare categories.

df["city"].value_counts(dropna=False)

7. Outliers. A common rule from your statistics course: values more than 1.5 x IQR beyond the quartiles.

q1, q3 = df["minutes"].quantile([0.25, 0.75])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
outliers = df[(df["minutes"] < low) | (df["minutes"] > high)]

An outlier is a question, not an error: it may be a typo (180 instead of 18), a real extreme (a delivery stuck in a storm) or a different kind of record (a bulk order). Investigate before removing anything, and never remove outliers just because they spoil a result.

8. Distributions and relationships. Mean vs median per group, and how key variables relate:

df.groupby("city")["minutes"].agg(["count", "median", "mean"])
df[["minutes", "rating"]].corr()

A mean far above the median signals a long right tail. A correlation is a starting point for a question, not a conclusion.

9. Time coverage (if there are dates). Are there gaps or a partial last month?

df.set_index("date").resample("MS").size()

Make it a function

Wrapping the checks in a function means you can run the same EDA on next month's file, or on a subset:

def quick_eda(df):
    return {
        "rows": len(df),
        "missing": df.isna().sum().to_dict(),
        "duplicates": int(df.duplicated().sum()),
    }

A worked example

Delivery data, 11 rows: order_id, city, minutes, rating.

• Shape: 11 rows, 4 columns; one row = one order. But order_id 7 appears twice: one exact duplicate.
• Missing: minutes 1, rating 2.
• Ranges: minutes runs from 28 to 180. With Q1 = 30.5 and Q3 = 35.75, the upper fence is about 43.6, so 180 is an outlier. Its rating is 1, so it looks like a real bad delivery rather than a typo; keep it, but report the median as well as the mean.
• Groups: Hull's mean delivery time is pulled far above its median by that one order.

Each finding becomes either a cleaning step (drop the duplicate) or a note in the write-up (one extreme delivery in Hull).

Common mistakes

• Computing results before the checklist. Every later number depends on it.
• Only looking at the mean. Always compare with the median and look at the extremes.
• Deleting outliers automatically. Investigate and document; decide case by case.
• Ignoring where missing values are. Missing data concentrated in one group biases group comparisons.
• Not writing anything down. EDA findings belong in the notebook as markdown notes.

Quick recap

• Check shape, types, missing, duplicates, ranges, categories, outliers, distributions and time coverage.
• Use the 1.5 x IQR rule to flag outliers, then investigate them.
• Compare mean with median; a large gap means skew.
• Put the checks in a function so they are repeatable, and record every finding.

Try it

In the Do tab you will write eda_report(df, column) that returns the core checklist numbers for the delivery dataset, including an IQR outlier count.