Design, test, refactor, review, estimate and run software the way professional teams do.

Software Engineering Practices

## Why it matters

Once software is running on servers you do not sit in front of, the only way to know what it is doing is the signals it sends out. **Observability** is how well you can answer questions about a running system from those signals: is it healthy, is it slow, for whom, and why? Teams with good observability find problems before users report them and fix them in minutes. Teams without it guess. The three standard signals are **logs**, **metrics** and **traces**.

## The three signals

| Signal | What it is | Good for | Example |
|---|---|---|---|
| **Logs** | Timestamped records of individual events | Details of one request or error | "checkout failed for order 991: card declined" |
| **Metrics** | Numbers aggregated over time | Trends, dashboards, alerts | requests per second, error rate, p95 latency |
| **Traces** | The path of one request through several services, with timings | Finding which step is slow | API 40 ms -> auth 5 ms -> database 310 ms |

They work together: a metric alert tells you the error rate jumped, a trace shows which service is failing, and the logs for those requests tell you why.

## Structured logging

Compare two log lines:

```text
User 118 checkout failed after 412ms: card declined
{"level":"error","time":"2026-10-23T09:14:02.120Z","msg":"checkout failed","userId":118,"durationMs":412,"reason":"card_declined","requestId":"r-7f3a"}
```

The first is easy to read but hard to search: to find all card declines over 400 ms you would need a fragile regular expression. The second is **structured**: one JSON object per line with consistent field names. Log tools can filter (`reason = card_declined AND durationMs > 400`), group and count it directly.

A minimal structured logger is only a few lines. Inject the output and the clock so it can be tested:

```javascript
function createLogger(write, clock, base = {}) {
  function log(level, msg, fields = {}) {
    const entry = { level, time: clock.now().toISOString(), msg, ...base, ...fields };
    write(JSON.stringify(entry));
  }
  return {
    info: (msg, fields) => log("info", msg, fields),
    error: (msg, fields) => log("error", msg, fields),
  };
}

const logger = createLogger(console.log, { now: () => new Date() }, { service: "orders" });
logger.info("order created", { orderId: 991, totalCents: 4599 });
```

Libraries such as **pino** or **winston** do the same with more features (log levels, fast serialisation, redaction). Good habits whichever tool you use:

- **Levels:** `debug` for development detail, `info` for normal events, `warn` for something odd but handled, `error` for failures that need attention.
- **Context fields:** request id, user id, route, duration. A **request id** generated at the edge and passed through every service lets you follow one request everywhere.
- **Never log secrets or sensitive data:** passwords, tokens, full card numbers, and usually raw email addresses. Redact them in the logger so a careless call cannot leak them.

## Metrics

Metrics answer "how much, how often, how fast" cheaply, because they store numbers rather than every event. The common types:

- **Counter:** only goes up. `http_requests_total`, `orders_created_total`.
- **Gauge:** goes up and down. Queue length, memory in use.
- **Histogram:** counts values in buckets, so you can compute percentiles. Request duration.

Metrics carry **labels** (also called tags or dimensions), such as `route` and `status`:

```text
http_requests_total{route="/checkout",status="500"} 37
```

Keep label values to a small, fixed set. A label like `userId` creates one time series per user (a "high cardinality" problem) and can overload a metrics system.

Two well-known starting sets: the **RED** method for services (Rate, Errors, Duration) and the **four golden signals** from Google's SRE book (latency, traffic, errors, saturation). Alert on symptoms users feel, such as error rate and latency, rather than on every internal number.

## Traces

A trace follows one request across services. Each step is a **span** with a start time, duration and attributes; spans nest to show which call waited on which. **OpenTelemetry** is the open, vendor-neutral standard for producing logs, metrics and traces, and most observability tools can receive its data. You rarely write tracing code by hand: OpenTelemetry's automatic instrumentation creates spans for HTTP calls and database queries.

## Common mistakes

- Logging unstructured sentences that nobody can search.
- Logging everything at `info` until the useful lines drown, or logging nothing until there is an outage.
- Putting passwords, tokens or personal data in logs.
- Using unbounded values (user ids, URLs with ids) as metric labels.
- Alerts that fire constantly, so people learn to ignore them.

## Recap

- Logs for details, metrics for trends and alerts, traces for where time goes.
- Structured JSON logs with levels, a request id and redaction of secrets.
- Counters, gauges and histograms with a small set of labels; start with RED or the golden signals.

## Try it

In the **Do** tab, build a small structured logger with redaction and a request counter, then use both in a request handler.


Add structured JSON logs with redaction and a labelled request counter to a request handler.

Logging, metrics, tracing

Observability vs. APM vs. Monitoring

12 Logging BEST Practices in 12 minutes

Take a small Node project (for example the API from Back-End Web APIs or the `debug-lab` folder from this course) with at least one test that `npm test` runs. Push it to a new public GitHub repository.

Search the code for hard-coded secrets (`grep -rn "key\|secret\|password" --exclude-dir=node_modules .`). Move any value into `process.env`, add `.env` to `.gitignore`, and commit a `.env.example` with the variable names only. Make the app throw a clear error at startup if a required variable is missing.

Run `npm audit`. Note the number of findings by severity, run `npm audit fix` for compatible upgrades, run your tests again, and write down any finding you cannot fix and why it is or is not reachable in your app.

In the repository settings, open **Code security** (called "Advanced Security" in some accounts) and enable Dependabot alerts, Dependabot security updates, secret scanning and push protection.

Add `.github/dependabot.yml` that checks the `npm` ecosystem weekly, commit it, and check the **Insights > Dependency graph > Dependabot** tab for its first run.

Add the CI workflow from the reading as `.github/workflows/ci.yml` (npm ci, npm test, npm audit --audit-level=high). Push and confirm the run is green in the **Actions** tab.

Open a pull request that breaks a test on purpose and confirm CI marks it as failing. Close the PR without merging.

Clean up: if you do not want to keep the repository, delete it in **Settings > Danger zone**. Free accounts have no charges for these features on public repositories, but check your billing page if you used a private one.

## Why it matters

Most security incidents in ordinary web apps do not come from brilliant attackers. They come from a handful of everyday mistakes: an API key committed to a public repository, a dependency with a known vulnerability that nobody updated, user input passed straight into a query. Secure development means building habits and automated checks so those mistakes are caught before they reach production. This lesson focuses on two of the most common problems, **secrets** and **dependencies**, and the free tools that catch them. The Back-End Web APIs course covered input validation, password hashing and the OWASP API Top 10; the same ideas apply here.

## Secrets: never in the code

A **secret** is anything that grants access: API keys, database passwords, signing keys for tokens, cloud credentials. The rule is simple: **secrets never go in source code or in Git**.

```javascript
// Wrong: the key is in the code, in every clone, and in Git history forever.
const payments = createClient("live-key-1234-abcd");

// Right: read it from the environment, and fail loudly if it is missing.
const key = process.env.PAYMENT_API_KEY;
if (!key) throw new Error("PAYMENT_API_KEY is not set");
const payments = createClient(key);
```

- In development, keep values in a `.env` file that is listed in `.gitignore`, and commit a `.env.example` with the variable names but no values.
- In production, set them in your host's environment settings or a secret manager.
- Give each environment its own secrets, and give each key the smallest permissions it needs.

**If a secret is ever committed, treat it as leaked**, even if you delete the line in the next commit. The old commit is still in history, and bots scan public GitHub for keys within minutes. The fix is to **rotate** it: create a new key in the provider's dashboard, deploy the new one, and revoke the old one. Rewriting Git history afterwards is optional clean-up, not the fix.

GitHub helps here. **Secret scanning** detects known key formats in repositories, and **push protection** blocks a push that contains one before it lands. Both are free for public repositories.

## Dependencies: the code you did not write

A typical Node project has a few direct dependencies and hundreds of indirect ones. Each is code that runs with your app's permissions. Risks include **known vulnerabilities** (published as CVEs and security advisories) and **supply-chain attacks**, where a package or one of its maintainers' accounts is compromised and a malicious version is published.

Habits that keep the risk manageable:

- **Commit the lock file** (`package-lock.json`) and install with `npm ci` in CI, so every build uses exactly the reviewed versions.
- **Audit regularly.** `npm audit` lists known vulnerabilities in your installed tree with their severity. `npm audit fix` applies compatible upgrades.

```bash
npm audit --audit-level=high   # exits non-zero if any high or critical advisory exists
```

- **Automate updates.** **Dependabot** (free on GitHub) opens pull requests that bump vulnerable or outdated dependencies. Your CI runs the tests on each one, so you can merge with confidence.
- **Add dependencies deliberately.** Before installing, check that a package is maintained, widely used and actually needed. A few lines of your own code can beat a tiny package with a large dependency tree.
- **Read the advisory before panicking.** Not every finding is exploitable in your app; a vulnerability in a build tool that never touches user input is lower priority than one in your request parser. Fix the reachable ones first.

## Put the checks in CI

Checks that depend on someone remembering will be forgotten. Make them part of every pull request. A minimal GitHub Actions workflow, free for public repositories:

```yaml
# .github/workflows/ci.yml
name: CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npm test
      - run: npm audit --audit-level=high
```

## Other secure-coding basics

- **Validate input** at every boundary and use parameterised queries, never string-built SQL.
- **Escape output** in HTML (frameworks like React do this by default; avoid bypassing it).
- **Check permissions on the server** for every request, not just in the UI.
- **Fail closed:** if a security check errors, deny access.
- **Keep secrets and personal data out of logs and error messages.**

## Common mistakes

- "It's a private repo, so a key in the code is fine." Private repos get cloned, forked, shared and sometimes made public.
- Deleting a leaked key from the code without rotating it.
- Ignoring audit output for months until there are hundreds of findings.
- Merging Dependabot PRs without tests, or never merging them at all.
- Installing a package with a misspelled name (typosquatting).

## Recap

- Secrets live in environment variables or a secret manager, never in Git; a committed secret must be rotated.
- Lock files, `npm ci`, `npm audit` and Dependabot keep dependencies under control.
- Put tests and audits in CI so every pull request is checked automatically.

## Try it

In the **Do** tab, set up secret handling, dependency scanning and a CI workflow on a free GitHub repository.


Keep secrets out of Git, keep dependencies patched, and put security checks in CI.

Secure development

How to fix vulnerabilities automatically with Dependabot

Learning about secret scanning on GitHub

3.4 Hiding API Keys with Environment Variables (dotenv) and Pushing Code to GitHub

Assign roles (or play them all yourself): incident commander, investigator and communications lead. Write one sentence on what each role does and does not do.

Choose a severity level for the incident using the table in the reading, and justify it in one sentence.

Write the first two status updates the communications lead would post (one at 10:10, one at 10:30), each with what is known, what is being done, and the time of the next update.

Write the timeline in UTC from the scenario, adding the times you would expect for declaring the incident and for each status update.

Work out the duration and describe the impact in a sentence (who was affected and how), without inventing exact user numbers you do not have.

Ask "why?" at least four times starting from "latency went up" and list at least three contributing factors. Write them about the system and process, not about people.

Write at least three action items, each specific, with an owner role and a due date, for example a migration checklist step or a staging test with production-sized data.

Review the draft: remove any sentence that blames a person, and check that every action item would prevent or detect this whole class of problem, not just this one index.

## Why it matters

Every team that runs software has incidents: an outage, a bad deploy, a full disk, a payment provider that stops responding. What separates mature teams is not that they never have incidents but how they handle them: they restore service quickly, communicate clearly while it is happening, and learn from it afterwards so the same failure does not happen again. The learning step is the **postmortem** (also called an incident review or retrospective), and the most effective kind is **blameless**.

## During the incident

An **incident** is any unplanned event that harms users or puts them at risk. Most teams use **severity levels**, for example:

| Severity | Meaning | Example |
|---|---|---|
| SEV1 | Major outage or data at risk; all hands | Nobody can log in |
| SEV2 | Significant feature broken or badly degraded | Checkout fails for 20% of users |
| SEV3 | Minor problem with a workaround | Export button slow for large accounts |

A calm response follows a few steps:

1. **Detect.** Ideally an alert on error rate or latency fires before users notice. Otherwise a support ticket or a teammate reports it.
2. **Declare and assign roles.** Someone says "this is an incident" and opens a channel. One person is the **incident commander**, who coordinates and makes decisions but does not debug. Others investigate. Someone handles **communication** with users and stakeholders.
3. **Mitigate first.** Restore service before finding the root cause. Roll back the last deploy, turn off a feature flag, scale up, or fail over. A rollback in five minutes beats a perfect fix in two hours.
4. **Communicate on a schedule.** Short updates at regular intervals ("Checkout errors since 14:05. We have rolled back the 14:00 deploy and errors are falling. Next update 14:45."), even when there is no news.
5. **Keep a timeline.** Note the time of every observation and action in the incident channel. You will need it for the postmortem.
6. **Resolve** when users are no longer affected, then schedule the review.

## The blameless postmortem

A postmortem is a written review of what happened, why, and what will change. **Blameless** means it assumes everyone acted reasonably given the information, tools and pressure they had at the time. The question is never "who made the mistake?" but "how did our system make this mistake easy to make, and hard to catch?"

This is not about being nice. It is practical. If people are punished for mistakes, they hide them, and the organisation stops learning. If an engineer can run one command that deletes production data, the real problem is the system that allowed it, and the fix is a safeguard, not a reprimand.

A typical postmortem has these sections:

```markdown
# Postmortem: Checkout errors, 2026-10-23

**Severity:** SEV2 · **Duration:** 14:05–14:41 UTC (36 min) · **Status:** Action items open

## Summary
A deploy changed the tax service URL. 22% of checkouts failed for 36 minutes.

## Impact
~1,300 failed checkout attempts; ~400 users affected; no data loss.

## Timeline (UTC)
- 14:00 Deploy 2026.10.23-2 released
- 14:05 Error-rate alert fires for /checkout
- 14:12 Incident declared; Ana is incident commander
- 14:31 Rollback started
- 14:41 Error rate back to normal; incident resolved

## Root cause and contributing factors
- The tax service URL was moved into a new config variable, set in staging but not production.
- The app started without it and only failed when checkout called the tax service.
- The deploy checklist has no step for new config variables.

## What went well
- The alert fired within 5 minutes.
## What went poorly
- Rollback took 19 minutes to start because the runbook was out of date.

## Action items
| Action | Owner | Due |
|---|---|---|
| Fail at startup if required config is missing | Ravi | 2026-10-30 |
| Add config check to the deploy pipeline | Mei | 2026-11-06 |
| Update rollback runbook and practise it | Ana | 2026-11-06 |
```

Notes on writing a good one:

- **Contributing factors, plural.** Real incidents rarely have one cause. Asking "why?" several times in a row (the **5 Whys**) helps you move from "a variable was missing" to "nothing checks for missing variables".
- **Action items are specific, owned and dated.** "Be more careful" is not an action item. "Fail at startup if a required variable is missing" is.
- **Prefer fixes that prevent or detect** the whole class of problem over fixes for this one instance.
- **Describe actions neutrally.** Write "the deploy was released at 14:00", not "Ravi carelessly deployed".
- **Share it widely** and review it in a meeting. Track action items until they are done; a postmortem whose actions never happen is just a story.

## Common mistakes

- Debugging for an hour before trying a rollback.
- Everyone investigating and nobody coordinating or communicating.
- Postmortems that name and shame, or stop at "human error".
- Action items with no owner or due date, or so many that none get done.
- Skipping the postmortem for "small" incidents that keep repeating.

## Recap

- Detect, declare, assign roles, mitigate first, communicate on a schedule, keep a timeline.
- Blameless postmortems ask how the system allowed the failure, not who caused it.
- Summary, impact, timeline, contributing factors, what went well and poorly, and owned, dated action items.

## Try it

In the **Do** tab, run a short incident simulation and write a blameless postmortem for it.


Respond to an incident in clear roles, mitigate first, and write a blameless postmortem with owned action items.

Incidents and postmortems

Blameless Postmortem Culture In Software Engineering

How to run a Post-Mortem meeting, step-by-step

Add structured logs and metrics, handle secrets and dependencies securely, and respond to incidents with blameless postmortems.