Monitoring and alerting

Why it matters

A pipeline can deliver a change in minutes, but you still need to know whether the system is working for users right now. Monitoring collects measurements from your systems; alerting tells a person when something needs attention. Done badly, alerting either misses real outages or wakes people at 3 a.m. for nothing until they start ignoring it. Site reliability engineering (SRE), the practice Google popularized, gives a clear method: decide what "working" means from the user's point of view, measure that, and alert on it.

Signals: what to measure

For a request-serving service, start with the four golden signals:

Use percentiles for latency, not averages. An average of 200 ms can hide the 5% of users waiting 4 seconds. p95 = 600 ms means 95% of requests finished within 600 ms.

Metrics are one of three kinds of telemetry. Logs record individual events, and traces follow one request across services. Together they are called observability. Common free and open-source tools are Prometheus (metrics), Grafana (dashboards), Loki (logs) and OpenTelemetry (instrumentation); every cloud has a built-in equivalent such as CloudWatch, Azure Monitor or Google Cloud Monitoring.

SLIs, SLOs and SLAs

• An SLI (service level indicator) is a measurement of user experience, usually a ratio of good events to all events: "the share of requests that succeed in under 500 ms".
• An SLO (service level objective) is your target for an SLI over a window: "99.5% of requests succeed in under 500 ms, measured over 30 days".
• An SLA (service level agreement) is a contract with customers that has consequences, such as refunds. SLAs are set looser than SLOs so you notice trouble before you owe anyone money.

No service should aim for 100%. Users cannot tell 99.99% from 100%, the last fraction is enormously expensive, and a target of perfection means you can never change anything.

Error budgets

The gap between 100% and the SLO is the error budget: how much unreliability you are allowed.

SLO: 99.5% availability over 30 days
30 days = 30 x 24 x 60 = 43,200 minutes
Error budget = 0.5% x 43,200 = 216 minutes of full outage (or the same share of failed requests)

SLO: 99.9%  -> 43.2 minutes per 30 days

The budget turns reliability into a decision tool. While budget remains, the team ships features. When it is nearly spent, the team slows releases and works on reliability. Product and engineering agree on this policy in advance, so nobody argues during an incident.

Alerting that people trust

Good alerts follow a few rules:

1. Alert on symptoms, not causes. Page on "users are getting errors" (the SLO is at risk), not on "CPU is at 85%". High CPU with happy users is not an emergency; low CPU with a broken database is.
2. Every page must be actionable and urgent. If the right response is "look at it tomorrow", it is a ticket, not a page.
3. Use burn rate. The burn rate is how fast you are spending the error budget compared with spending it evenly over the window. A burn rate of 1 uses exactly the whole budget in 30 days. A common starting point from Google's SRE workbook: page when the burn rate is about 14.4 over the last hour (2% of a 30-day budget gone in one hour), and open a ticket for slower burns such as a burn rate of 1 over three days.
4. Link every alert to a runbook: what the alert means, the first checks to make, and how to mitigate.
5. Review alerts regularly. Delete or fix any alert that fired without needing action.

Here is a Prometheus-style example for the error ratio:

groups:
  - name: hello-devops-slo
    rules:
      - alert: HighErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{job="hello",code=~"5.."}[1h]))
            /
            sum(rate(http_requests_total{job="hello"}[1h]))
          ) > (14.4 * 0.005)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "hello-devops is burning its 30-day error budget fast"
          runbook: "https://example.com/runbooks/hello-devops-errors"

14.4 * 0.005 is the burn rate times the allowed error ratio for a 99.5% SLO: page if more than 7.2% of requests failed over the past hour.

For a quick start without any of that, a synthetic check (an uptime monitor that requests your URL every minute) catches "the site is down" and is easy to run yourself.

Common mistakes

• Dozens of cause-based alerts (CPU, disk, memory) that page for things users never notice.
• Averages instead of percentiles for latency.
• SLOs chosen by wishful thinking instead of from current performance and user needs.
• Alerts with no runbook, sent to a shared inbox nobody watches.

Quick recap

• Measure latency, traffic, errors and saturation; use percentiles.
• SLIs measure user experience; SLOs set targets; SLAs are contracts.
• The error budget is 100% minus the SLO; burn rate shows how fast you spend it.
• Page only for urgent, user-facing symptoms, each with a runbook.

Try it

The Do tab has you define an SLO for your Compose app, compute its error budget and set up a free local uptime monitor that alerts when the app goes down.