Monitoring and billing alerts

Why it matters

If you cannot see what your system is doing, you find out about problems from angry users or a bill. Monitoring tells you when something is broken, slow or expensive, ideally before anyone else notices. In the cloud you also monitor money: cost is a metric like CPU, and it needs alerts just as much.

The core idea: metrics, logs, traces, alarms

• Metrics are numbers over time: CPU percentage, requests per minute, error count, estimated charges. Cloud services publish many metrics automatically. On AWS they go to Amazon CloudWatch; Azure has Azure Monitor; Google has Cloud Monitoring.
• Logs are timestamped records of events: application output, web server access logs, and the audit log of who called which API (AWS CloudTrail, Azure Activity Log, Google Cloud Audit Logs).
• Traces follow one request across several services (covered in the DevOps & Cloud Delivery course).
• Alarms watch a metric and act when it crosses a threshold for a period: send an email or chat message, or trigger an automatic action.

Health checks

A health check asks "is this working?" on a schedule.

• Status checks: EC2 runs system and instance status checks on every VM and publishes the StatusCheckFailed metric.
• Load balancer health checks: the load balancer calls a path such as /healthz on each server every few seconds and stops sending traffic to servers that fail.
• External checks: a probe outside your network loads your public URL, as a user would. This catches problems (DNS, certificates, a whole region) that internal checks miss.

A good health endpoint is cheap and honest:

# Flask example
@app.get("/healthz")
def healthz():
    db_ok = ping_database(timeout_seconds=1)
    return ({"ok": db_ok}, 200 if db_ok else 503)

Writing alarms people trust

An alarm should mean "a human needs to act". For each one decide:

Metric:     StatusCheckFailed for instance practice-vm
Statistic:  Maximum over 1 minute
Threshold:  >= 1 for 2 consecutive periods
Action:     email the on-call address (via an SNS topic)
Runbook:    "Check the console, reboot, check system log, replace from image"

Alarm on symptoms users feel (errors, latency, health checks failing) before causes (CPU at 80%). Too many noisy alarms train people to ignore them.

Billing alerts, revisited

In the first module you created a budget. Now make cost monitoring a habit:

• Budgets alert on actual or forecast spend for the month, for the whole account or filtered by service or tag.
• Cost anomaly detection (AWS Cost Anomaly Detection, and similar tools on other clouds) flags unusual spend patterns, such as a service that suddenly costs ten times more than usual.
• Tags such as project=practice and owner=you let you see which project is costing what and find forgotten resources.
• Cost Explorer (or Azure Cost Management, Google Billing reports) shows spend by service and day. Open it weekly.

Remember: a budget alert notifies, it does not stop resources. The cleanup habit is still yours.

Worked example: a forgotten resource

You get a budget email: forecast spend is 14 US dollars against a 1-dollar budget. In Cost Explorer, grouped by service, "EC2 - Other" stands out. Grouped by usage type, it shows NAT gateway hours. You left a NAT gateway from a networking tutorial in another region. You delete it, then add a habit: tag everything with project and check all regions in the console after each session. Without the alert, that would have run all month.

Common mistakes

• Alarms with no action, or sending to an inbox nobody reads.
• Only internal checks, so a DNS or certificate failure goes unnoticed.
• Logging secrets or personal data.
• Keeping debug-level logs forever and paying to store them; set retention periods.
• Treating a budget alert as a spending cap.

Quick recap

• Metrics, logs, traces and alarms answer "what is happening and when must a human act?"
• Health checks run at three levels: status checks, load balancer checks and external probes.
• Budgets, anomaly detection, tags and a weekly cost review catch forgotten resources early.

Try it

The Do tab has you add an alarm and a cost check to your account, then clean up.