Safe releases

Why it matters

Most outages are caused by changes: a new release, a configuration edit, a migration. You cannot stop changing things, so the goal is to make each change small, observable and reversible. A team that can undo a bad release in two minutes can afford to release many times a day. A team whose rollback takes an afternoon will release rarely, in big risky batches.

This lesson covers the two main tools for safe releases, canary deployments and feature flags, and how to plan a rollback before you need it.

Separate deploy from release

Two words that sound alike mean different things:

• Deploy: put new code on servers.
• Release: let users experience the new behaviour.

When you separate them, you can deploy code with a new feature switched off, check that nothing else broke, and then switch the feature on for a few users. Feature flags make that possible.

Canary deployments

A canary sends a small share of real traffic to the new version while the rest stays on the old one. You compare the two using metrics. If the canary is healthy, you increase its share in stages; if not, you send all traffic back to the old version.

A canary plan answers five questions before the release starts:

1. Stages and wait times. For example 5% for 15 minutes, 25% for 15 minutes, 50% for 30 minutes, then 100%.
2. Metrics. Usually the error rate (share of HTTP 5xx responses), latency (p95 or p99), and one business signal such as successful checkouts or log-ins.
3. Stop conditions. Concrete thresholds, written down: "abort if the canary's 5xx rate is more than 1 percentage point above the baseline for 5 minutes, or p95 latency is more than 20% higher".
4. Rollback. The exact command or button, who may press it and how long it takes.
5. Owner and communication. Who watches the dashboards, where status is posted, who is told when it is done.

Why compare with the baseline instead of a fixed number? Because traffic and errors vary by time of day. The old version, receiving the same kind of traffic at the same moment, is the fairest comparison.

Tools such as Argo Rollouts, Flagger, AWS CodeDeploy, Cloud Run traffic splitting and most service meshes can shift traffic in stages and abort automatically when metrics fail. Even with automation, the plan has to exist first: a tool cannot guess your thresholds.

Feature flags

A feature flag is a condition in code whose value comes from configuration at run time:

if (flags.isEnabled("new-checkout", { userId })) {
  return renderNewCheckout();
}
return renderOldCheckout();

Flags let you:

• Ship unfinished work to production switched off, so branches stay short.
• Turn a feature on for staff, then 1% of users, then everyone.
• Switch a broken feature off in seconds without a deploy. This is the kill switch.

Flags have costs. Every flag doubles the paths through that code, so test both, give each flag an owner and an expiry date, and delete it once the feature is fully on. Old flags left in code are a known source of confusing bugs.

Rollback, rehearsed

A rollback you have never practised is a hope, not a plan. Good rollbacks are:

• Fast: redeploy the previous image tag (which is why immutable SHA tags matter), switch blue-green traffic back, or turn a flag off.
• Rehearsed: try it in staging, timed, before the release that needs it.
• Data-aware: if the release ran a database migration, rolling back the code must still work with the new schema. Use the expand-and-contract pattern from the previous lesson.

Sometimes rolling forward (shipping a quick fix) is right, but decide that deliberately, and only if the fix is small and the impact is limited. Under pressure, rolling back is usually the safer default.

A short worked plan

Release: hello-devops 4b1e9d0 (new greeting endpoint). Stages: 5% for 15 min, 25% for 15 min, 100%. Metrics: 5xx rate, p95 latency, from the service dashboard. Abort if: canary 5xx rate is 1 point above baseline for 5 min, or p95 is 20% above baseline. Rollback: route 100% to 3f9c2a1 (tested in staging, takes about 1 minute); flag new-greeting off as a first step. Owner: Sam (release), Priya (watching dashboards). Status in #releases at each stage.

Common mistakes

• Starting a canary without written stop conditions, then arguing about whether a spike "counts".
• Watching only technical metrics while the business metric quietly drops.
• Canary stages too short to collect meaningful traffic.
• Releasing many unrelated changes at once, so you cannot tell which one broke.
• Flags that live forever.

Quick recap

• Separate deploy from release; feature flags give you a kill switch.
• A canary plan lists stages, metrics, stop conditions, the rollback and owners.
• Compare the canary with the baseline, not with a fixed number.
• Rehearse the rollback and keep migrations backward compatible.

Try it

The Do tab has you write a canary and rollback plan for your hello-devops app and rehearse the rollback locally with Docker.