Ship and run cloud software safely: containers, CI/CD pipelines, Terraform with cost guardrails, SLOs, alerting and incident response.

DevOps & Cloud Delivery

Add `test_app.py` from the Read tab and run `python -m unittest -v` locally until it passes.

Create `.github/workflows/ci.yml` with the workflow from the Read tab. Run `git status` and confirm `.env` is not listed before committing.

Create a new public repository on GitHub (no README), add it as `origin` and push your `main` branch.

Open the Actions tab and watch the run: `test`, then `build`, then `smoke-test` should go green. Open each job's log and find the image tag (the commit SHA).

Create a branch, change `greeting` so the test fails, push it and open a pull request. Confirm `test` goes red and `build` is skipped, so nothing is delivered.

Fix the code on the branch, push again, confirm the checks go green, and merge the pull request. Confirm the run on `main` pushes a new image tagged with the merge commit SHA.

In Settings > Branches, add a branch protection rule (or ruleset) for `main` that requires the `test` check to pass before merging.

Clean up: in your GitHub profile's Packages tab, open `hello-devops` and delete the package versions you do not need (or the whole package), and check Settings > Billing for Actions usage.

## Why it matters

Without automation, releasing software depends on someone remembering every step: run the tests, build the package, copy it to the server, restart the service. People skip steps when they are busy, and that is when outages happen. A **pipeline** turns those steps into code that runs the same way on every change.

**Continuous integration (CI)** means every push and pull request is built and tested automatically, so problems show up within minutes, while the change is still fresh in the author's mind. **Continuous delivery (CD)** extends that: every change that passes is packaged and ready to release, and releasing is a routine, low-drama step. (Continuous *deployment* goes one step further and releases every passing change automatically.)

## The core idea: stages that gate each other

A typical pipeline has stages, and each one runs only if the previous one passed:

1. **Checkout** the exact commit.
2. **Test**: unit tests, linting, type checks. Fast feedback first.
3. **Build** an artifact: a container image, tagged with the commit SHA.
4. **Deliver or deploy** that artifact, usually only from the main branch.
5. **Verify**: a smoke test against what was deployed.

A failure at any stage stops the run and marks the commit red, so a broken change never reaches users.

## GitHub Actions vocabulary

GitHub Actions runs pipelines defined in YAML files under `.github/workflows/`.

| Term | Meaning |
|---|---|
| Workflow | One YAML file; triggered by events |
| Event | What starts it: `push`, `pull_request`, a schedule, a manual run |
| Job | A set of steps on one fresh runner (VM). Jobs run in parallel unless linked with `needs` |
| Step | A shell command (`run`) or a reusable action (`uses`) |
| Runner | The machine that executes a job, for example `ubuntu-latest` |

Cost: standard GitHub-hosted runners are free for public repositories, and private repositories on the Free plan get a monthly quota of included minutes. Quotas change, so check GitHub's billing docs for Actions (checked on 2026-10-23) and watch the usage page in your account settings.

## A worked example

Add a test for the `greeting` function from the Docker lesson:

```python
# test_app.py
import unittest

from app import greeting


class GreetingTest(unittest.TestCase):
    def test_greeting(self):
        self.assertEqual(greeting("CI"), "Hello, CI!")


if __name__ == "__main__":
    unittest.main()
```

Then the workflow:

```yaml
# .github/workflows/ci.yml
name: ci
on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m unittest -v

  build:
    needs: test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: |
          OWNER="${GITHUB_REPOSITORY_OWNER,,}"
          echo "IMAGE=ghcr.io/$OWNER/hello-devops:${GITHUB_SHA}" >> "$GITHUB_ENV"
          docker build -t "ghcr.io/$OWNER/hello-devops:${GITHUB_SHA}" .
      - name: Push image (main only)
        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
        run: |
          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
          docker push "$IMAGE"

  smoke-test:
    needs: build
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: read
    steps:
      - name: Run the published image and check it answers
        run: |
          OWNER="${GITHUB_REPOSITORY_OWNER,,}"
          IMAGE="ghcr.io/$OWNER/hello-devops:${GITHUB_SHA}"
          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
          docker run -d --name app -p 8000:8000 "$IMAGE"
          curl --fail --retry 10 --retry-delay 1 --retry-all-errors http://localhost:8000
```

How it fits together:

- `on` runs the workflow for pull requests and for pushes to `main`.
- `needs: test` means the image is built only after tests pass; `needs: build` chains the smoke test.
- The image tag is the **commit SHA**, so you always know exactly which code an image contains.
- The `if:` conditions make pull requests run tests and a build, but only `main` publishes. That stops unreviewed code from being delivered.
- `GITHUB_TOKEN` is a short-lived token GitHub creates for every run. The `permissions` blocks give it the least access each job needs. No password is stored anywhere.
- GitHub Container Registry requires lowercase image names, hence `${GITHUB_REPOSITORY_OWNER,,}`.

In a real system the last job would deploy the image to a cloud service (for example by updating a Cloud Run service or a Kubernetes Deployment) and then smoke-test the live URL. The structure stays the same.

## Secrets in pipelines

- Never commit credentials, keys or `.env` files. Store them as **repository or environment secrets**; GitHub masks them in logs.
- Prefer short-lived credentials. `GITHUB_TOKEN` is one; for clouds, use **OIDC federation** so the workflow exchanges a GitHub-signed token for temporary cloud credentials instead of storing long-lived access keys.
- Pull requests from forks do not receive your secrets, by design. Be careful with the `pull_request_target` event, which does.
- Pin third-party actions to a version (`@v4`), or to a full commit SHA for stronger supply-chain safety.

## Common mistakes

- **Slow pipelines.** If CI takes 40 minutes, people batch changes and stop trusting it. Run fast checks first and cache dependencies.
- **Flaky tests ignored.** A test that fails randomly trains everyone to click "re-run". Fix or quarantine it.
- **Deploying from any branch.** Gate delivery on `main` (and on review via branch protection).
- **Rebuilding per environment.** Build once, then promote the same image; the next lesson explains why.

## Quick recap

- CI tests every change; CD keeps every passing change releasable.
- A workflow has events, jobs, steps and runners; `needs` orders jobs.
- Tag artifacts with the commit SHA and publish only from `main`.
- Use `GITHUB_TOKEN`, least-privilege `permissions` and secrets, never committed credentials.

## Try it

The Do tab has you push the `hello-devops` project to GitHub and watch this pipeline go red and then green.

Build a GitHub Actions pipeline that tests every change, builds a container image and delivers it from main only.

Pipelines

DevOps CI/CD Explained in 100 Seconds

How to use GitHub Actions | GitHub for Beginners

## Why it matters

Testing a change on a developer laptop is not the same as running it next to real infrastructure, real configuration and real traffic. Teams therefore run several copies of their system, called **environments**, and move each change through them before users see it. Getting this design right is a core part of a cloud engineer's job: too few environments and bugs reach users; too many and releases slow to a crawl while the cloud bill grows.

## The common environments

| Environment | Purpose | Who uses it | Data |
|---|---|---|---|
| Local / dev | Write and debug code | One developer | Fake or seeded |
| Preview | One short-lived copy per pull request, so reviewers can click through a change | Reviewers, designers | Seeded test data |
| Staging | A production-like copy for final checks, integration and load tests | The team, QA | Test or anonymized data |
| Production | The real system | Users | Real data |

Preview environments are created automatically when a pull request opens and deleted when it closes. Many platforms (Vercel, Netlify, Render, and Kubernetes setups with tools such as Argo CD) do this out of the box. Their short lifetime keeps cost down; forgetting to delete them is a classic source of surprise bills.

Staging is only useful if it resembles production: same container images, same infrastructure code, same kinds of services, smaller in size. This is called **environment parity**. A staging environment that runs a different database version or skips the load balancer will miss exactly the bugs it exists to catch.

## Build once, promote the artifact

The most important rule: **build an artifact once and promote that same artifact** from environment to environment. If you rebuild for each environment, the image that reaches production is not the one you tested. Dependencies may have changed, a base image may have been updated, or a build flag may differ.

```text
commit 3f9c2a1
   |
   v
CI builds image  hello-devops:3f9c2a1   (once)
   |
   +--> preview    runs hello-devops:3f9c2a1
   +--> staging    runs hello-devops:3f9c2a1  -> integration tests pass
   +--> production runs hello-devops:3f9c2a1  -> after approval
```

What differs between environments is **configuration**, not code: database URLs, feature flag settings, log levels, replica counts. Provide it at deploy time through environment variables and a secret store, following the twelve-factor app principle of keeping config in the environment.

**Promotion** means moving that artifact to the next environment. It can be:

- Automatic: every green build on `main` deploys to staging.
- Gated: production requires a manual approval, a change window or passing checks. GitHub Actions supports this with **environments** that have protection rules such as required reviewers, and each environment can have its own secrets.
- Branch-based: some teams fast-forward a `production` branch to a tested commit and let the platform deploy that branch. The principle is the same: production gets a commit that already passed everywhere else.

## Deployment strategies

How you replace the old version matters as much as where:

- **Recreate**: stop the old version, start the new one. Simple, but users see downtime.
- **Rolling update**: replace instances a few at a time (the Kubernetes default). No downtime, but old and new versions run side by side for a while.
- **Blue-green**: run the new version (green) next to the current one (blue) at full size, test it, then switch all traffic at the load balancer. Rollback is a switch back. It needs double capacity during the release.
- **Canary**: send a small share of traffic (say 5%) to the new version, watch its metrics, then increase. The next lesson plans one.

## Databases are the hard part

Code rolls back easily; data does not. If version 2 renames a column and you roll back to version 1, version 1 breaks. The safe pattern is **expand and contract**: first add the new column while keeping the old one (both versions work), migrate the data, switch the code, and only remove the old column in a later release. Every migration should be backward compatible with the version currently running.

## Common mistakes

- Rebuilding the artifact for production.
- Staging that has drifted from production, so it no longer predicts anything.
- Sharing one database between staging and production, or copying real customer data into test environments without anonymizing it.
- Preview environments that never get deleted.
- Storing production secrets where staging jobs can read them. Scope secrets per environment.

## Quick recap

- Preview, staging and production each have a job; staging must resemble production.
- Build once, then promote the same immutable artifact; only configuration changes.
- Gate production with approvals or checks.
- Choose a strategy (rolling, blue-green, canary) and keep database changes backward compatible.

## Try it

This lesson has no hands-on activity. Use the quiz in the Do tab to test your environment design.

Design preview, staging and production environments and promote one tested artifact through them.

Environments and promotion

Multi-stage deployments in GitHub Actions

Top 5 Most-Used Deployment Strategies

Pick a small change (for example a new greeting text) and write the release goal in one sentence in a file called `RELEASE-PLAN.md`.

Write the canary stages with traffic shares and wait times (for example 5%, 25%, 100%).

Choose three metrics: error rate, a latency percentile, and one user-facing signal. Note where each would be viewed.

Write concrete stop conditions with numbers and a time window, compared with the baseline version.

Add a feature flag to the plan: name it, say who can switch it, and when it will be deleted from the code.

Write the rollback: the previous image tag, the exact command, who may run it, and the expected time to recover.

Rehearse: build the current version as `hello-devops:v1` and the changed one as `hello-devops:v2`, run v2 with `docker run -d --name hello -p 8000:8000 hello-devops:v2`, then roll back by running `docker rm -f hello` and starting v1 the same way. Time how long it takes and confirm the old greeting returns.

Clean up with `docker rm -f hello` and `docker image rm hello-devops:v1 hello-devops:v2`, then commit `RELEASE-PLAN.md` with the measured rollback time.

## Why it matters

Most outages are caused by changes: a new release, a configuration edit, a migration. You cannot stop changing things, so the goal is to make each change **small, observable and reversible**. A team that can undo a bad release in two minutes can afford to release many times a day. A team whose rollback takes an afternoon will release rarely, in big risky batches.

This lesson covers the two main tools for safe releases, **canary deployments** and **feature flags**, and how to plan a rollback before you need it.

## Separate deploy from release

Two words that sound alike mean different things:

- **Deploy**: put new code on servers.
- **Release**: let users experience the new behaviour.

When you separate them, you can deploy code with a new feature switched off, check that nothing else broke, and then switch the feature on for a few users. Feature flags make that possible.

## Canary deployments

A **canary** sends a small share of real traffic to the new version while the rest stays on the old one. You compare the two using metrics. If the canary is healthy, you increase its share in stages; if not, you send all traffic back to the old version.

A canary plan answers five questions before the release starts:

1. **Stages and wait times.** For example 5% for 15 minutes, 25% for 15 minutes, 50% for 30 minutes, then 100%.
2. **Metrics.** Usually the error rate (share of HTTP 5xx responses), latency (p95 or p99), and one business signal such as successful checkouts or log-ins.
3. **Stop conditions.** Concrete thresholds, written down: "abort if the canary's 5xx rate is more than 1 percentage point above the baseline for 5 minutes, or p95 latency is more than 20% higher".
4. **Rollback.** The exact command or button, who may press it and how long it takes.
5. **Owner and communication.** Who watches the dashboards, where status is posted, who is told when it is done.

Why compare with the baseline instead of a fixed number? Because traffic and errors vary by time of day. The old version, receiving the same kind of traffic at the same moment, is the fairest comparison.

Tools such as Argo Rollouts, Flagger, AWS CodeDeploy, Cloud Run traffic splitting and most service meshes can shift traffic in stages and abort automatically when metrics fail. Even with automation, the plan has to exist first: a tool cannot guess your thresholds.

## Feature flags

A **feature flag** is a condition in code whose value comes from configuration at run time:

```javascript
if (flags.isEnabled("new-checkout", { userId })) {
  return renderNewCheckout();
}
return renderOldCheckout();
```

Flags let you:

- Ship unfinished work to production switched off, so branches stay short.
- Turn a feature on for staff, then 1% of users, then everyone.
- Switch a broken feature off in seconds without a deploy. This is the **kill switch**.

Flags have costs. Every flag doubles the paths through that code, so test both, give each flag an owner and an expiry date, and delete it once the feature is fully on. Old flags left in code are a known source of confusing bugs.

## Rollback, rehearsed

A rollback you have never practised is a hope, not a plan. Good rollbacks are:

- **Fast**: redeploy the previous image tag (which is why immutable SHA tags matter), switch blue-green traffic back, or turn a flag off.
- **Rehearsed**: try it in staging, timed, before the release that needs it.
- **Data-aware**: if the release ran a database migration, rolling back the code must still work with the new schema. Use the expand-and-contract pattern from the previous lesson.

Sometimes rolling **forward** (shipping a quick fix) is right, but decide that deliberately, and only if the fix is small and the impact is limited. Under pressure, rolling back is usually the safer default.

## A short worked plan

> **Release:** hello-devops `4b1e9d0` (new greeting endpoint).
> **Stages:** 5% for 15 min, 25% for 15 min, 100%.
> **Metrics:** 5xx rate, p95 latency, from the service dashboard.
> **Abort if:** canary 5xx rate is 1 point above baseline for 5 min, or p95 is 20% above baseline.
> **Rollback:** route 100% to `3f9c2a1` (tested in staging, takes about 1 minute); flag `new-greeting` off as a first step.
> **Owner:** Sam (release), Priya (watching dashboards). Status in #releases at each stage.

## Common mistakes

- Starting a canary without written stop conditions, then arguing about whether a spike "counts".
- Watching only technical metrics while the business metric quietly drops.
- Canary stages too short to collect meaningful traffic.
- Releasing many unrelated changes at once, so you cannot tell which one broke.
- Flags that live forever.

## Quick recap

- Separate deploy from release; feature flags give you a kill switch.
- A canary plan lists stages, metrics, stop conditions, the rollback and owners.
- Compare the canary with the baseline, not with a fixed number.
- Rehearse the rollback and keep migrations backward compatible.

## Try it

The Do tab has you write a canary and rollback plan for your `hello-devops` app and rehearse the rollback locally with Docker.

Plan a canary release with clear metrics, stop conditions, a feature-flag kill switch and a rehearsed rollback.

Safe releases

Canary Deployment: Safer Releases Explained

What are Feature Flags?

Automate testing and delivery with GitHub Actions, promote one artifact through environments, and release safely with canaries and feature flags.