Load, clean, reshape and explore real data with Python and pandas, then turn the results into clear findings.

Intro to Data Analysis with Python

Write two questions you would ask the requester to learn what decision the analysis supports, and write a plausible answer to each.

Name the stakeholder, how technical they are, and the deadline.

Define the main measure as a formula, including what is excluded (for example: loans per month = completed e-book checkouts, excluding renewals and staff test accounts).

Write 2 to 4 specific, comparative questions, headline question first. Each should name the measure, the population, the period and what it is compared with.

For each question, list the data you would need: table or file, columns, date range and any known problems.

Write at least two assumptions and one item that is out of scope.

Mark any question that is secretly causal ("did X cause Y?") and rewrite it as a comparison you can honestly answer.

Write a three-line message you would send to play the plan back to the requester.

## Why it matters

Analysis requests rarely arrive as clear questions. They sound like "Can you look into sales?", "Why are the results worse lately?" or "Is the new process working?" If you start coding straight away, you will produce lots of numbers and no answer. The most valuable skill in this module is turning a vague request into a few **specific, answerable questions** before touching the data.

## What makes a question good

A good analysis question is:

- **Specific about the measure.** "Revenue", "on-time rate", "mean yield", not "performance".
- **Specific about the population and time.** Which customers, sites or samples, over which period?
- **Comparative.** Against what? Last year, a target, another group, before and after a change.
- **Answerable with data you have, or can get.**
- **Tied to a decision.** If the answer were X, what would someone do differently?

| Vague | Specific |
|---|---|
| How are sales doing? | Did Q3 revenue per region change compared with Q3 last year, and which region changed most? |
| Is the new process working? | Is mean batch yield on line L2 higher in the 8 weeks after the October change than in the 8 weeks before? |
| Why are deliveries slow? | Which cities and hours have a median delivery time over 40 minutes in September, and what share of orders do they cover? |

## A method: from request to questions

**1. Ask about the decision.** "What will you do with the answer?" or "What prompted this?" The reply often narrows the work by half. A manager who asks "how are sales doing" may really be deciding where to put two new staff.

**2. Name the stakeholder and the deadline.** Who reads the result, how technical are they, and when do they need it? A rough answer on Friday may beat a perfect one next month.

**3. Define the measures precisely.** Write the formula: "on-time = delivered within 30 minutes of the promised time; rate = on-time orders / all delivered orders, excluding cancellations". Ambiguous definitions are the most common source of arguments about results.

**4. Write 2 to 4 questions,** each in the specific form above. Order them: the headline question first, then the ones that explain it.

**5. Check the data exists.** For each question list the table, the columns, the date range and any known problems. If a question needs data you do not have, say so early.

**6. Write down assumptions and what is out of scope.** "Refunds excluded; online orders only; no causal claim, this is a before/after comparison."

**7. Play it back.** Send the questions and definitions to the requester in a few lines and ask "Is this what you need?" Ten minutes here saves days of rework.

## Association is not causation

Many requests contain a hidden causal question: "Did the campaign increase sales?" A before/after comparison or a correlation can show that two things moved together, not that one caused the other. Seasonality, a price change, or a different customer mix could explain the same pattern. Be honest in how you phrase the question ("Did sales differ...?") unless you have a proper experiment, which the `regression-experimental-design` course covers.

## A worked example

Request from an operations manager: "Deliveries feel slow lately. Can you look into it?"

After a five-minute conversation:

- **Decision:** whether to add drivers in some cities for the evening peak.
- **Stakeholder:** operations manager, needs a one-page summary by Thursday.
- **Measure:** delivery time = delivered timestamp minus order timestamp, in minutes; use the median because a few very late orders skew the mean.
- **Questions:**
  1. Has the median delivery time in September changed compared with July and August?
  2. Which city and hour combinations have a median over 40 minutes in September?
  3. What share of September orders fall in those slow combinations?
- **Data:** `orders.csv` (order and delivered times, city), July to September. Known issue: about 2% of rows lack a delivered time.
- **Out of scope:** driver-level performance; causes (traffic, weather) unless obvious.

That is a plan someone can say yes or no to, and an analysis you can finish.

## Common mistakes

- **Starting with the data.** Exploring without a question is fine for 30 minutes, not for a week.
- **Undefined measures.** "Active user" or "late" without a formula leads to two analysts getting two answers.
- **Too many questions.** Four focused questions beat fifteen shallow ones.
- **Promising causes from observational data.**
- **Not playing the questions back** to the person who asked.

## Quick recap

- Ask what decision the analysis supports, who reads it and by when.
- Define each measure as a formula, including exclusions.
- Write 2 to 4 specific, comparative, answerable questions and check the data exists.
- Record assumptions and scope, then confirm with the requester.

## Try it

The Do tab gives you a vague request and walks you through turning it into a one-page analysis plan.


Turn a vague analysis request into specific, answerable questions with defined measures and scope.

Asking good questions

How to Ask the Right Questions as a Data Analyst | Google Data Analytics Certificate

## Why it matters

Exploratory data analysis (EDA) is the structured first look at a dataset before you trust it. It answers: What is in here? What is broken? What is surprising? Skipping EDA is how analysts report averages inflated by a single typo, or "trends" caused by a month of missing data. A short, repeatable checklist catches most of these problems in minutes.

## The EDA checklist

Work through these in order and write down what you find.

**1. Shape and meaning.** How many rows and columns? What is one row (one order, one sample, one day)? Does the count match what you were told?

```python
df.shape
df.head()
df.sample(5, random_state=1)   # random rows, reproducible
```

**2. Types.** Are numbers numeric, dates datetime, IDs text?

```python
df.dtypes
```

**3. Missing values.** How many per column, and are they random or concentrated (one site, one month)?

```python
df.isna().sum()
df.isna().mean().round(3)      # share missing per column
```

**4. Duplicates.** Whole-row duplicates and duplicate keys.

```python
df.duplicated().sum()
df["order_id"].duplicated().sum()
```

**5. Ranges and summaries.** Do minimums and maximums make sense? Negative ages, delivery times of 0 or 900 minutes, percentages over 100?

```python
df.describe()
```

**6. Categories.** Spelling variants, unexpected codes, rare categories.

```python
df["city"].value_counts(dropna=False)
```

**7. Outliers.** A common rule from your statistics course: values more than 1.5 x IQR beyond the quartiles.

```python
q1, q3 = df["minutes"].quantile([0.25, 0.75])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
outliers = df[(df["minutes"] < low) | (df["minutes"] > high)]
```

An outlier is a question, not an error: it may be a typo (180 instead of 18), a real extreme (a delivery stuck in a storm) or a different kind of record (a bulk order). Investigate before removing anything, and never remove outliers just because they spoil a result.

**8. Distributions and relationships.** Mean vs median per group, and how key variables relate:

```python
df.groupby("city")["minutes"].agg(["count", "median", "mean"])
df[["minutes", "rating"]].corr()
```

A mean far above the median signals a long right tail. A correlation is a starting point for a question, not a conclusion.

**9. Time coverage** (if there are dates). Are there gaps or a partial last month?

```python
df.set_index("date").resample("MS").size()
```

## Make it a function

Wrapping the checks in a function means you can run the same EDA on next month's file, or on a subset:

```python
def quick_eda(df):
    return {
        "rows": len(df),
        "missing": df.isna().sum().to_dict(),
        "duplicates": int(df.duplicated().sum()),
    }
```

## A worked example

Delivery data, 11 rows: `order_id, city, minutes, rating`.

- **Shape:** 11 rows, 4 columns; one row = one order. But `order_id` 7 appears twice: one exact duplicate.
- **Missing:** `minutes` 1, `rating` 2.
- **Ranges:** `minutes` runs from 28 to 180. With Q1 = 30.5 and Q3 = 35.75, the upper fence is about 43.6, so 180 is an outlier. Its rating is 1, so it looks like a real bad delivery rather than a typo; keep it, but report the median as well as the mean.
- **Groups:** Hull's mean delivery time is pulled far above its median by that one order.

Each finding becomes either a cleaning step (drop the duplicate) or a note in the write-up (one extreme delivery in Hull).

## Common mistakes

- **Computing results before the checklist.** Every later number depends on it.
- **Only looking at the mean.** Always compare with the median and look at the extremes.
- **Deleting outliers automatically.** Investigate and document; decide case by case.
- **Ignoring where missing values are.** Missing data concentrated in one group biases group comparisons.
- **Not writing anything down.** EDA findings belong in the notebook as markdown notes.

## Quick recap

- Check shape, types, missing, duplicates, ranges, categories, outliers, distributions and time coverage.
- Use the 1.5 x IQR rule to flag outliers, then investigate them.
- Compare mean with median; a large gap means skew.
- Put the checks in a function so they are repeatable, and record every finding.

## Try it

In the Do tab you will write `eda_report(df, column)` that returns the core checklist numbers for the delivery dataset, including an IQR outlier count.


Run an EDA checklist on a dataset: shape, types, missing values, duplicates, ranges, outliers and distributions.

Exploratory analysis

Exploratory Data Analysis in Pandas | Python Pandas Tutorials

Exploratory Data Analysis with Pandas Python

Write the main question in one sentence, and the definition of its key measure.

Rerun your notebook from a fresh kernel (restart and run all) and copy the numbers you need from that run.

Write a one- or two-sentence headline that answers the question with the key number and a comparison.

Write two or three lines of context: data source, period, number of rows and the measure definition.

Write three to five finding bullets, each one claim with a rounded number.

Add at most one chart per key finding, each with a title that states the finding (or describe the chart you would add).

Write a recommendation or next step that follows from the evidence, using non-causal wording unless you ran an experiment.

Write a short caveats and method section: cleaning decisions, missing data, limits of the period and a link or path to the notebook.

Read it as the stakeholder: cut any sentence or number that does not help a decision, and check the page fits on one side.

## Why it matters

An analysis only creates value when someone reads it and acts. Most decision-makers will read one page, often just the first paragraph. Writing a clear findings summary is therefore as important as the code behind it, and it is what hiring managers look for in a portfolio project.

## The one-page structure

**1. Headline (one or two sentences).** The answer to the main question, with the key number. Write it so that someone who reads nothing else still gets the point.

> Median delivery time rose from 31 to 37 minutes between August and September. Almost all of the increase comes from evening orders in Leeds and York.

**2. Context (two or three lines).** The question, the data (source, period, rows) and the definition of the main measure.

**3. Key findings (three to five bullets).** Each bullet is one claim, backed by a number and, where useful, one chart.

- Evening (5 to 8 pm) orders in Leeds and York had a median of 46 minutes in September, up from 33 in August.
- Those orders are 22% of September volume.
- Other cities and times were unchanged (within 1 minute).

**4. Recommendation or next step.** What the reader could do, framed by the evidence: "Consider adding evening drivers in Leeds and York; a two-week trial would show whether delivery times recover."

**5. Caveats and method (short).** What could make the findings wrong: missing data, a short period, no causal design, cleaning decisions (for example "1 duplicate row removed; 2% of orders lacked a delivery time and were excluded"). Link to the notebook for detail.

## Writing well about numbers

- **Lead with the answer,** not with how you got it. Method goes last.
- **Round sensibly.** "37 minutes", not "36.8421 minutes"; "22%", not "21.96%".
- **Give comparisons.** A number alone means little: "37 minutes, up from 31".
- **Use the reader's language.** "Evening orders" rather than `hour_bucket == 3`.
- **Say what is uncertain.** "Based on four weeks of data" or "the difference is small relative to week-to-week variation".
- **Avoid causal words** (caused, drove, because of) unless your design supports them; prefer "coincided with", "is associated with".
- **One chart per key finding at most,** each with a title that states the finding (from module 2).

## Turning notebook output into findings

A useful habit: under each analysis cell in your notebook, write one markdown sentence starting "This shows...". At the end, those sentences are your draft bullets. Then sort them by importance and cut any that do not help the reader decide.

## Reproducibility

Before you send the summary:

- Restart the kernel and run all; check the numbers in the text match the output.
- Put the data source, the date it was pulled and the notebook link in the method section.
- If you are sharing a portfolio project, include a short README with how to run it, and never publish private or personal data.

## A worked example of bad vs good

**Bad:** "I loaded the data and cleaned it, then did a groupby on city and hour. The mean for Leeds hour 18 was 47.3. There were some outliers. Charts attached."

**Good:** "Evening deliveries in Leeds and York are the main reason delivery times rose in September (median 46 minutes vs 33 in August; 22% of orders). Elsewhere times were stable. Adding evening drivers in those two cities is the most targeted fix. Caveat: one month of data; weather was not checked."

The good version answers the question, quantifies it, says what to do and admits the limits, in four sentences.

## Common mistakes

- **Narrating the process** instead of stating findings.
- **Too many numbers.** Keep only those that support a decision.
- **Hiding caveats,** or burying the headline under them.
- **Mismatched numbers** between text and charts after a late change to the notebook.
- **Overclaiming causes** from observational data.

## Quick recap

- Headline answer first, then context, 3 to 5 findings with numbers, a recommendation and short caveats.
- Round, compare, use plain language and avoid unsupported causal claims.
- Draft bullets from "This shows..." notes in your notebook.
- Rerun everything before sending, and record source and method.

## Try it

The Do tab walks you through writing a one-page findings summary for an analysis you have done in this course or on a public dataset.


Write a one-page findings summary that leads with the answer, backs it with numbers and states its limits.

Writing up findings

7 Effective Tips for Presenting Data at Work!

Telling Stories with Data in 3 Steps (Quick Study)

Run a small analysis from start to finish: turn a vague request into answerable questions, explore the data with an EDA checklist, and write up the findings on one page.