Monitoring What Actually Matters

Introduction¶

Monitoring everything produces dashboards nobody reads and alerts nobody trusts. A few well-chosen frameworks narrow it to the signals that predict user pain.

The four golden signals¶

For any user-facing service, watch:

1. Latency — how long requests take. Split successful vs failed (a fast 500 shouldn't flatter your latency graph). Track percentiles (p50, p95, p99), not averages.
2. Traffic — demand: requests/sec, transactions/sec, concurrent users.
3. Errors — rate of failed requests (5xx, timeouts, wrong answers). Explicit failures and implicit ones (200 with a broken body).
4. Saturation — how full the system is: CPU, memory, connection pool, queue depth, disk I/O. The resource that runs out first.

If you only have time to set up four graphs per service, these are the four.

RED (for request-driven services)¶

• Rate — requests per second
• Errors — failed requests per second
• Duration — latency distribution

Simple, works for every microservice, easy to templatise across dozens of services.

USE (for resources)¶

For every resource (CPU, disk, NIC, pool):

• Utilisation — % busy
• Saturation — queued work it can't service yet
• Errors — error events

USE catches "the box is fine on CPU% but the run queue is 40 deep" — saturation often bites before utilisation hits 100%.

SLIs and SLOs¶

• SLI (indicator) — a metric that reflects user experience: successful requests / total requests, or requests faster than 300ms / total.
• SLO (objective) — a target for the SLI over a window: "99.9% of requests succeed, measured over 28 days".
• Error budget — 100% − SLO. 99.9% over 28 days ≈ 40 minutes of allowed failure. If you've spent the budget, stop shipping features and fix reliability; if you have budget to spare, you can take more risk.

Alert on error-budget burn rate (are we going to blow the SLO?), not on individual metric blips. A fast burn (will exhaust budget in an hour) pages; a slow burn opens a ticket.

Black-box vs white-box¶

• Black-box — probe the service from outside like a user does (HTTP check, synthetic transaction, DNS resolution). Catches "the whole thing is down" and DNS/TLS/LB problems your internal metrics miss. This is what tools like the uptime check do.
• White-box — metrics from inside (Prometheus, statsd, app instrumentation). Tells you why.

You want both: black-box to know users are affected, white-box to diagnose.

Dashboards for 3 a.m.¶

• One screen per service, top-to-bottom: is it up (black-box), the four golden signals, then dependencies.
• Annotate deploys on the time axis — "it broke at 02:14" + "deploy at 02:12" answers most incidents instantly.
• Show the SLO line on the error/latency graphs so "is this bad?" is obvious.
• Link the runbook from the dashboard.
• Resist the 40-panel dashboard. If a panel has never helped during an incident, remove it.

Verification and troubleshooting¶

• Metrics exist but nobody looks — too many dashboards. Consolidate to one per service + one fleet overview.
• Averages hiding pain — p99 latency can be 10× p50. Always graph percentiles; a rise in p99 with flat p50 means a subset of users are hurting.
• SLO always green but users complain — the SLI doesn't capture their experience (measuring server-side latency, not including the CDN; counting a slow-but-200 as success). Refine the SLI.
• Saturation alert fires constantly — utilisation-based, not saturation-based; or the resource is genuinely undersized. Alert on queue depth / wait time, not %.
• Can't tell if an alert matters — it's not tied to an SLI. Every page should map to "this threatens an SLO" or "this is a hard failure".
• New service has no monitoring — templatise RED + a black-box check as part of the service scaffold so it's never an afterthought.

Related tools and reading¶

• On-site: Uptime / status check, HTTP Header Analyzer, DNS lookup.
• Related posts: Alert fatigue and how to fix it, Understanding Linux load average.