Alert Fatigue and How to Fix It

Introduction¶

Alert fatigue is when responders become desensitised because most alerts are non-actionable. It's a safety problem: the real incident arrives buried in noise and gets the same shrug as the nightly false alarm.

Symptoms¶

• On-call routinely acknowledges alerts without investigating.
• A shared "known noisy, ignore it" list exists.
• Alerts fire and resolve themselves before anyone looks.
• The runbook for an alert is "wait, it usually clears".
• New team members are told which alerts "don't really matter".

If any of these are true, fix alerting before adding more monitoring.

The principle: every alert is actionable¶

An alert should mean: a human needs to do something now that automation can't. If the answer to "what do I do about this?" is "nothing" or "wait", it shouldn't page. Options for a non-actionable signal:

• Delete it if it's not useful.
• Downgrade to a dashboard / ticket if it's useful context but not urgent.
• Automate the response if the action is always the same (restart, scale, clear a queue).
• Fix the underlying flakiness if it's a real but transient condition.

A remediation program¶

1. Measure¶

Pull 30–90 days of alert history. For each alert rule, count: fires, how many were acknowledged with no action, how many self-resolved, mean time to resolve. Sort by volume.

2. Attack the top offenders¶

The distribution is always long-tailed — a handful of rules produce most of the noise. For each:

• Self-resolves in < 5 min → add a for: / first_notification_delay so it must persist before paging.
• Fires at a threshold that's within normal operating range → move the threshold, or alert on rate-of-change / sustained breach instead of an instantaneous value.
• Duplicates another alert (10 hosts, same root cause) → add dependencies / grouping so one incident = one page.
• Never actionable → delete or downgrade.

3. Alert on symptoms, not causes¶

Page on "users are affected" (error rate, latency SLO burn, checkout failures), not on every possible cause (CPU 85%, disk 70%, one pod restarted). Causes go on dashboards you look at after a symptom alert. This alone often cuts rules by half.

A useful frame (Google SRE's): alert on the four golden signals — latency, traffic, errors, saturation — at the service boundary.

4. Deduplicate and group¶

Use the alertmanager/notification layer to group by incident: group_by: [alertname, cluster, service], a group_wait and group_interval so a burst becomes one notification with a count.

5. Route by urgency¶

• Page (wake someone): symptom alerts with user impact, now.
• Ticket / chat: needs attention this week (cert expiring in 20 days, disk trending full).
• Dashboard only: context.

Most things that currently page belong in the second or third bucket.

6. Review regularly¶

Every incident retro: "did the alerting help or hurt?" Monthly: re-run the volume report, kill the new top offenders. Treat a paging alert that led to no action as a bug to be fixed.

Verification and troubleshooting¶

• Track alerts per on-call shift as a metric. A healthy number is single digits per week for most teams; if it's dozens per night, keep cutting.
• Track actionability rate — of alerts that paged, what fraction led to a human action. Aim for > 80%.
• "But if I delete it and something breaks?" — that's what symptom alerts are for. If a deleted cause-alert would have caught something a symptom alert misses, keep it as a ticket, not a page.
• Threshold tuning keeps not sticking — you're tuning static thresholds on a metric with daily/weekly seasonality. Use a window (avg_over_time), a percentage-of-baseline, or anomaly detection.
• One noisy service dominates — its alerts may be a proxy for genuine instability. The fix is reliability work on that service, not just muting it.

Related tools and reading¶

• On-site: Uptime / status check.
• Related posts: Monitoring what actually matters, Nagios notifications and escalations.