Alertmanager Routing and Silences

Introduction¶

Prometheus decides what is wrong (rules fire alerts); Alertmanager decides who hears about it, grouped how, and how often. It dedupes, groups, routes, silences, and inhibits.

The routing tree¶

route:
  receiver: default-slack
  group_by: [alertname, cluster, service]
  group_wait: 30s              # wait to batch alerts that fire together
  group_interval: 5m           # how often to send an updated batch for a group
  repeat_interval: 4h          # re-send an unresolved alert after this long

  routes:
    - matchers: [severity="critical"]
      receiver: pagerduty
      group_wait: 0s
      continue: true           # ALSO evaluate later routes / send to Slack too
    - matchers: [severity="critical"]
      receiver: critical-slack
    - matchers: [team="payments"]
      receiver: payments-slack
      routes:
        - matchers: [severity="warning"]
          receiver: payments-tickets
    - matchers: [alertname=~"Watchdog|DeadMansSwitch"]
      receiver: heartbeat        # to a dead-man's-switch monitor

• An alert enters at the root and walks down; it matches a child route if its labels match the matchers. First matching child wins unless continue: true (then it also continues to siblings).
• If no child matches, the alert uses the parent's receiver.
• Child routes inherit group_by, timings, etc. unless overridden.

Grouping¶

group_by: [alertname, cluster, service] — all alerts sharing those label values form one group and produce one notification (with all the alerts listed). So "50 pods down on cluster X" is one page, not 50.

• group_by: [...] — group by these labels.
• group_by: ['...'] with '...' (literal three dots) — don't group, one notification per alert. Rarely what you want.
• group_wait — after the first alert in a new group, wait this long for more before sending (batches a cascade).
• group_interval — minimum time between notifications for the same group when its contents change (a new alert joins).
• repeat_interval — re-notify about an unresolved group after this long. Too short = nag fatigue; 4h is a common floor for pages.

Inhibition¶

Suppress alerts that are consequences of a bigger alert:

inhibit_rules:
  - source_matchers: [alertname="NodeDown"]
    target_matchers: [alertname=~"NodeExporterDown|HighLatency|InstanceDown"]
    equal: [instance]           # only inhibit targets on the SAME instance

If NodeDown fires for instance=web-3, don't also page about NodeExporterDown on web-3 — it's the same event.

Silences vs inhibition vs mute times¶

• Silence — a manual, time-bounded mute for alerts matching a label matcher. "I'm doing maintenance on db-2 for 2 hours." Created in the UI or amtool.
• Inhibition — automatic, rule-based, "don't alert on the symptom while the cause is alerting."
• Mute time intervals — scheduled quiet periods on a route (mute_time_intervals), e.g. "don't send ticket-severity alerts on weekends". Use active_time_intervals for the inverse.

Dead man's switch¶

Add an always-firing alert (Watchdog) that routes to an external "if this stops arriving, something is broken" monitor (Healthchecks.io, PagerDuty heartbeat). If your whole monitoring stack dies, Prometheus stops sending Watchdog, and the external monitor pages you.

Verification and troubleshooting¶

amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml \
  severity=critical team=payments alertname=HighErrorRate      # which receiver?
amtool alert query
amtool silence add alertname=HighLatency instance=web-3 --duration=2h --comment="INC-123"
amtool silence query
curl -s localhost:9093/api/v2/alerts | jq

• No notifications at all — Prometheus isn't sending to Alertmanager (--web.enable-lifecycle / alerting.alertmanagers config, check the Prometheus "Status → Runtime" and prometheus_notifications_* metrics), the receiver's integration is misconfigured (bad Slack webhook / PagerDuty key — Alertmanager logs the send error), or a silence/mute is active.
• Alert routed to the wrong receiver — amtool config routes test with the alert's exact labels shows the path. A too-broad matchers on an earlier child caught it. Order and continue: matter.
• Getting paged for a symptom of a known outage — missing inhibition rule.
• Nag every 5 minutes — repeat_interval too low, or group_interval firing because the group keeps changing (flapping alerts join/leave). Fix the flapping alert (for: clause), raise the intervals.
• Resolved notifications are noisy — send_resolved: false on the receiver if you don't want them.
• Silence not working — the matcher doesn't match the alert's labels (Alertmanager matches on the alert's label set, including ones added by relabeling); amtool alert query shows the actual labels. Also silences are per-Alertmanager unless you run an HA cluster with gossip.
• HA Alertmanager sending duplicates — the cluster gossip isn't working (--cluster.peer config, port 9094); each instance thinks it's alone.

Related tools and reading¶

• On-site: YAML ↔ JSON converter, Regex Tester.
• Related posts: SLO burn-rate alerts, Alert fatigue and how to fix it.