Ship and run cloud software safely: containers, CI/CD pipelines, Terraform with cost guardrails, SLOs, alerting and incident response.

DevOps & Cloud Delivery

In a file `SLO.md`, write one availability SLI and one latency SLI for the web app as good-events-over-all-events ratios.

Choose an SLO for each over 30 days (for example 99.5%) and calculate the error budget in minutes. Show your arithmetic.

Write two alert rules in plain English: one page (fast budget burn) and one ticket (slow burn). Give each a one-paragraph runbook: what it means, first checks, how to mitigate.

Start your stack with `docker compose up -d` and start a monitor: `docker run -d --name uptime-kuma -p 3001:3001 louislam/uptime-kuma:1` (on Linux add `--add-host=host.docker.internal:host-gateway`).

Open http://localhost:3001, create a local admin account, and add an HTTP monitor for `http://host.docker.internal:8000` with a 20-second interval.

Stop the web service with `docker compose stop web`, wait for the monitor to turn red, and note how long detection took. Then run `docker compose start web` and watch it recover.

Compare the downtime you just caused with your error budget, and write one sentence in `SLO.md` about how much of the month's budget it would have used.

Clean up: `docker rm -f uptime-kuma`, `docker compose down`, and commit `SLO.md`.

## Why it matters

A pipeline can deliver a change in minutes, but you still need to know whether the system is working for users right now. **Monitoring** collects measurements from your systems; **alerting** tells a person when something needs attention. Done badly, alerting either misses real outages or wakes people at 3 a.m. for nothing until they start ignoring it. Site reliability engineering (SRE), the practice Google popularized, gives a clear method: decide what "working" means from the user's point of view, measure that, and alert on it.

## Signals: what to measure

For a request-serving service, start with the **four golden signals**:

| Signal | Question | Example metric |
|---|---|---|
| Latency | How long do requests take? | p95 and p99 response time |
| Traffic | How much demand is there? | Requests per second |
| Errors | How many requests fail? | Share of HTTP 5xx responses |
| Saturation | How full is the system? | CPU, memory, connection pool, queue depth |

Use **percentiles** for latency, not averages. An average of 200 ms can hide the 5% of users waiting 4 seconds. p95 = 600 ms means 95% of requests finished within 600 ms.

Metrics are one of three kinds of **telemetry**. **Logs** record individual events, and **traces** follow one request across services. Together they are called observability. Common free and open-source tools are Prometheus (metrics), Grafana (dashboards), Loki (logs) and OpenTelemetry (instrumentation); every cloud has a built-in equivalent such as CloudWatch, Azure Monitor or Google Cloud Monitoring.

## SLIs, SLOs and SLAs

- An **SLI** (service level indicator) is a measurement of user experience, usually a ratio of good events to all events: "the share of requests that succeed in under 500 ms".
- An **SLO** (service level objective) is your target for an SLI over a window: "99.5% of requests succeed in under 500 ms, measured over 30 days".
- An **SLA** (service level agreement) is a contract with customers that has consequences, such as refunds. SLAs are set looser than SLOs so you notice trouble before you owe anyone money.

No service should aim for 100%. Users cannot tell 99.99% from 100%, the last fraction is enormously expensive, and a target of perfection means you can never change anything.

## Error budgets

The gap between 100% and the SLO is the **error budget**: how much unreliability you are allowed.

```text
SLO: 99.5% availability over 30 days
30 days = 30 x 24 x 60 = 43,200 minutes
Error budget = 0.5% x 43,200 = 216 minutes of full outage (or the same share of failed requests)

SLO: 99.9%  -> 43.2 minutes per 30 days
```

The budget turns reliability into a decision tool. While budget remains, the team ships features. When it is nearly spent, the team slows releases and works on reliability. Product and engineering agree on this policy in advance, so nobody argues during an incident.

## Alerting that people trust

Good alerts follow a few rules:

1. **Alert on symptoms, not causes.** Page on "users are getting errors" (the SLO is at risk), not on "CPU is at 85%". High CPU with happy users is not an emergency; low CPU with a broken database is.
2. **Every page must be actionable and urgent.** If the right response is "look at it tomorrow", it is a ticket, not a page.
3. **Use burn rate.** The burn rate is how fast you are spending the error budget compared with spending it evenly over the window. A burn rate of 1 uses exactly the whole budget in 30 days. A common starting point from Google's SRE workbook: page when the burn rate is about 14.4 over the last hour (2% of a 30-day budget gone in one hour), and open a ticket for slower burns such as a burn rate of 1 over three days.
4. **Link every alert to a runbook**: what the alert means, the first checks to make, and how to mitigate.
5. **Review alerts regularly.** Delete or fix any alert that fired without needing action.

Here is a Prometheus-style example for the error ratio:

```yaml
groups:
  - name: hello-devops-slo
    rules:
      - alert: HighErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{job="hello",code=~"5.."}[1h]))
            /
            sum(rate(http_requests_total{job="hello"}[1h]))
          ) > (14.4 * 0.005)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "hello-devops is burning its 30-day error budget fast"
          runbook: "https://example.com/runbooks/hello-devops-errors"
```

`14.4 * 0.005` is the burn rate times the allowed error ratio for a 99.5% SLO: page if more than 7.2% of requests failed over the past hour.

For a quick start without any of that, a **synthetic check** (an uptime monitor that requests your URL every minute) catches "the site is down" and is easy to run yourself.

## Common mistakes

- Dozens of cause-based alerts (CPU, disk, memory) that page for things users never notice.
- Averages instead of percentiles for latency.
- SLOs chosen by wishful thinking instead of from current performance and user needs.
- Alerts with no runbook, sent to a shared inbox nobody watches.

## Quick recap

- Measure latency, traffic, errors and saturation; use percentiles.
- SLIs measure user experience; SLOs set targets; SLAs are contracts.
- The error budget is 100% minus the SLO; burn rate shows how fast you spend it.
- Page only for urgent, user-facing symptoms, each with a runbook.

## Try it

The Do tab has you define an SLO for your Compose app, compute its error budget and set up a free local uptime monitor that alerts when the app goes down.

Define SLIs, SLOs and an error budget for a service, then write alerts that page people only for user-facing problems.

Monitoring and alerting

SLIs, SLOs, SLAs, oh my! (class SRE implements DevOps)

SRE Golden Signals Explained

Write a one-page incident template in `INCIDENT.md`: severity table, roles (commander, operations, communications, scribe), the four-part status update format and the review headings from the Read tab.

Assign roles (or note that you hold them all), then start the stack with `docker compose up -d` and start a timer.

Cause the failure: run `docker compose stop db`, then load http://localhost:8000 and confirm it reports `db_reachable=False`.

As commander, declare the incident: choose a severity and write the first status update with the time, impact, current action and next update time.

As operations, investigate with `docker compose ps` and `docker compose logs db`, and write each finding in the scribe's timeline with a timestamp.

Mitigate with `docker compose start db`, confirm the page shows `db_reachable=True`, and post a resolved update.

Write a blameless review: impact, timeline, contributing factors, one thing that went well, and at least two action items with an owner and a date (for example "add a health-check alert for the database").

Clean up with `docker compose down` and commit `INCIDENT.md`.

## Why it matters

Every system fails eventually. What separates good teams from struggling ones is not whether incidents happen but how they are handled: how fast the right people are involved, whether someone is clearly in charge, whether users and stakeholders know what is going on, and whether the team learns enough to stop the same failure happening again. Cloud engineers are usually part of an **on-call rotation**, so incident skills are part of the job from early on.

## On-call basics

Being **on call** means you are the first responder for alerts during a shift, usually a week at a time. A healthy rotation has:

- **Enough people**: at least five or six per rotation, so each person is on call rarely enough to rest.
- **A primary and a secondary** responder, with clear **escalation**: if the primary does not acknowledge a page within a few minutes, it goes to the secondary, then to a manager.
- **Handoffs**: at the end of a shift, write down open issues, recent changes and anything flaky.
- **Runbooks** linked from every alert.
- **Fair compensation and time off** after a bad night. Burned-out responders make mistakes.
- A **paging tool** (PagerDuty, Opsgenie, incident.io, Grafana OnCall or a cloud equivalent) that handles schedules, escalation and acknowledgement.

## Severity levels

Teams classify incidents so everyone knows how much urgency to apply. One common scheme:

| Severity | Example | Response |
|---|---|---|
| SEV1 | Site down or data loss for many users | All hands, incident commander, updates every 15–30 min |
| SEV2 | Major feature broken, or degraded for many users | Incident commander, updates every 30–60 min |
| SEV3 | Minor problem with a workaround | Handled in normal hours through a ticket |

When unsure, declare the higher severity. It is easy to downgrade later and costly to realize too late that you needed more help.

## Incident roles

Many organizations borrow from the **Incident Command System** used by emergency services. In a serious incident, people take distinct roles:

- **Incident commander (IC)**: coordinates. Decides priorities, assigns tasks, keeps everyone focused. The IC deliberately does **not** debug; their job is the overall picture.
- **Operations lead / subject matter experts**: investigate and apply fixes, and report findings to the IC.
- **Communications lead**: posts regular status updates to stakeholders and the public status page, so engineers are not interrupted by "any news?" messages.
- **Scribe**: keeps a timeline of what was seen, decided and done, with times. This becomes the backbone of the review.

In a small team one person may hold two roles, but say out loud who holds which.

## The flow of an incident

1. **Detect**: an alert fires or a user reports a problem.
2. **Declare**: open an incident channel, set severity, name the IC.
3. **Mitigate first**: stop the user impact before finding the root cause. Roll back the last release, turn off a feature flag, fail over, add capacity. Understanding can wait; users cannot.
4. **Communicate**: status updates on a fixed rhythm, even when the update is "still investigating, next update at 14:30".
5. **Resolve**: confirm the metrics are back within SLO, and announce it.
6. **Review**: hold a post-incident review within a few days.

A useful status update has four parts:

```text
14:05 UTC (SEV2) Checkout errors for about 20% of users since 13:42.
Impact: some payments fail; retrying may work.
Current action: rolling back release 4b1e9d0, which deployed at 13:40.
Next update: 14:25 UTC or sooner if things change.
```

## Blameless post-incident reviews

A **blameless** review asks "how did our system and process allow this?", not "who messed up?" People who fear blame hide information, and hidden information makes the next incident worse. A review document usually has:

- Summary and impact (who was affected, for how long, how much error budget was used).
- A timeline from the scribe's notes.
- Contributing factors: technical (a missing test, a risky migration) and process (no canary, an alert that took 20 minutes to fire).
- What went well, as well as what went badly.
- **Action items** with an owner and a due date, focused on prevention and faster detection. "Be more careful" is not an action item; "add a canary stage with a 5xx stop condition" is.

## Common mistakes

- Everyone debugging and nobody coordinating or communicating.
- Hunting for the root cause while users are still affected, instead of mitigating first.
- Silence: stakeholders hear nothing for an hour and assume the worst.
- Reviews that name a culprit, or that produce action items nobody owns.
- Pages without runbooks, routed to a single exhausted person.

## Quick recap

- Healthy on-call has enough people, escalation, handoffs, runbooks and rest.
- Severity sets urgency; declare higher when unsure.
- Separate roles: commander, operations, communications, scribe.
- Mitigate first, update on a rhythm, and run blameless reviews with owned action items.

## Try it

The Do tab is a tabletop incident drill: you break your own Compose app, run the incident with named roles and write the review.

Run an incident with clear roles, severity levels, regular status updates and a blameless post-incident review.

On-call and incidents

How HashiCorp does Incident Management

Collect five cloud engineer or DevOps job postings for roles you would apply to, and count which clouds and tools (AWS, Azure, Google Cloud, Terraform, Kubernetes) they mention.

Choose one associate certification that matches the most common requirement, and open its official certification page from the table in the Read tab.

Download or open the official exam guide, list every domain with its weight, and note the current exam version and renewal period. Write "checked on" with today's date next to them.

Rate yourself 1–3 on each domain and order the domains by weight times gap.

Write a dated weekly plan for 6–10 weeks with hours per week, naming the provider's free official training for each of your weakest domains.

For each domain, write one hands-on lab you will do, with the cleanup step for each lab that uses cloud resources.

Confirm that a budget alert exists on your cloud practice account (from the cost guardrails lesson) and note its threshold in the plan.

Write a README outline for your portfolio repository that links the Compose stack, pipeline, Terraform guardrails, SLO and incident review from this course, and check its Git history for secrets with `git log -p | grep -i -E "password|secret|key"`.

## Why it matters

Cloud engineer job postings often list a cloud certification, and recruiters use them as a filter. Hiring managers, though, care more about whether you can do the work: build a pipeline, write Terraform, debug a broken deployment, explain a design. The best use of a certification is as a **structured study plan** that fills gaps in your knowledge and gives you a recognized signal, alongside a portfolio that proves you can apply it. A certificate without hands-on skill tends to fall apart in the first technical interview.

## The main associate-level options

Exam details, prices and versions change often, so always read the official page. Checked on 2026-10-23.

| Certification | Body | Focus | Official page |
|---|---|---|---|
| AWS Certified Solutions Architect – Associate | Amazon Web Services | Designing secure, resilient, cost-aware AWS architectures | [aws.amazon.com/certification](https://aws.amazon.com/certification/) |
| Microsoft Certified: Azure Administrator Associate | Microsoft | Running Azure identity, storage, compute and networking | [learn.microsoft.com/credentials](https://learn.microsoft.com/credentials/) |
| Google Cloud Associate Cloud Engineer | Google Cloud | Deploying and operating workloads on Google Cloud | [cloud.google.com/learn/certification](https://cloud.google.com/learn/certification) |
| HashiCorp Certified: Terraform Associate | HashiCorp | Terraform concepts, state, modules, workflow | [developer.hashicorp.com/certifications](https://developer.hashicorp.com/certifications) |
| Certified Kubernetes Administrator (CKA) | The Linux Foundation / CNCF | Running Kubernetes clusters (hands-on exam) | [training.linuxfoundation.org](https://training.linuxfoundation.org/certification/) |

Some useful distinctions:

- The three cloud certifications are **platform** certifications. Pick the cloud that appears most in the job postings you want. Skills transfer between clouds; the names of services differ.
- Terraform Associate and CKA are **tool** certifications that apply on any cloud. Terraform Associate pairs well with a cloud associate certification. The CKA is a performance-based exam in a live terminal and is generally considered harder than the multiple-choice exams.
- Each provider also has a **foundational** exam (AWS Cloud Practitioner, Azure Fundamentals, Google Cloud Digital Leader). If you have finished Cloud Fundamentals and this course, you can usually go straight to an associate exam.
- Certifications **expire** or need renewal, typically every two to three years. Check the renewal rules on the official page.

## Reading an exam guide

Every exam publishes an **exam guide** (also called a study guide or skills outline) that lists domains and their weights, for example "Design secure architectures: 30%". This is the most valuable free document you will read for the exam. Use it to:

1. List every domain and its weight.
2. Rate yourself 1–3 on each topic: 1 = never used it, 2 = read about it, 3 = have built it.
3. Spend your time in proportion to *weight times gap*: a heavily weighted topic you rated 1 comes first.

## A study plan that works

A realistic plan for someone who has finished this course is 6–10 weeks at 5–8 hours a week:

- **Weeks 1–2**: official free training from the provider (AWS Skill Builder, Microsoft Learn, Google Cloud Skills Boost, HashiCorp tutorials) for your weakest heavy domains.
- **Weeks 3–6**: hands-on labs for every domain. Build it, break it, fix it, destroy it. Use a free tier with a budget alert, and delete resources at the end of every session.
- **Weeks 7–8**: an official practice exam or sample questions. Review every wrong answer until you can explain why each option is right or wrong.
- **Final week**: re-read the exam guide, revisit weak spots, book the exam for a time when you are rested.

Two rules about integrity: never use "brain dumps" (leaked real exam questions). They violate the candidate agreement you sign and can get your certification revoked, and they do not teach you anything. And never share live exam content afterwards.

## Pair it with a portfolio

The projects from this course already form a strong portfolio: a containerized app with Compose, a CI/CD pipeline that tests, builds and publishes images, Terraform with tags and validation, an SLO document and an incident review. Put them in a public repository with a clear README that explains the design decisions, and make sure no secrets are anywhere in its history. In interviews, being able to walk through a real pipeline you built often counts for more than the certificate itself.

Beyond the platform: cloud engineer roles also expect hands-on Linux and networking experience, and many employers prefer a bachelor's degree in computer science or a related field, though experience and certifications can substitute. Internships, help-desk or junior operations roles and open-source contributions are common ways in.

## Common mistakes

- Collecting many foundational certificates instead of one associate certification plus a portfolio.
- Studying only from videos, with no hands-on practice.
- Leaving lab resources running after a study session.
- Ignoring the exam guide's weights.

## Quick recap

- Pick one associate certification that matches the jobs you want; read its official page.
- Turn the exam guide's domains into a weighted study plan.
- Practise hands-on on a free tier with a budget alert; clean up every session.
- Pair the certification with a portfolio of real, secret-free projects.

## Try it

The Do tab has you choose a certification and turn its official exam guide into a dated study plan.

Choose an associate-level cloud or infrastructure certification that fits real job postings and build a study plan from its official exam guide.

Associate certifications

How I Passed the AWS Solutions Architect Exam as a BEGINNER

Terraform Associate Exam Tips - Everything You Need to Pass

Measure what users experience with SLOs, alert on what matters, run incidents well, and plan an associate certification.