SLO Burn-Rate Alerts

Introduction¶

An SLO gives you an error budget (100% − target). Burn-rate alerting fires when you're consuming that budget fast enough to matter — not on every blip. It's the Google SRE workbook approach, and it dramatically cuts alert noise while catching real problems faster.

The concept¶

• SLO: 99.9% of requests succeed over 30 days.
• Error budget: 0.1% of requests. Over 30 days at, say, 10 req/s, that's a finite number of allowed failures.
• Burn rate: how fast you're spending it, relative to "spend it evenly over the window". Burn rate 1 = on track to exactly exhaust the budget at day 30. Burn rate 14.4 = you'll exhaust the whole 30-day budget in ~2 days.

Which burn rates to alert on¶

Multi-window, multi-burn-rate¶

The trick that prevents both false pages and slow detection: require a high burn rate over a short window AND a still-elevated burn rate over a longer window. The short window catches it fast; the long window confirms it's not a one-minute spike; and a "reset" is fast because the short window recovers quickly.

# fast-burn page: 14.4x over 1h AND over 5m
(
  error_ratio_1h > (14.4 * 0.001)
  and
  error_ratio_5m > (14.4 * 0.001)
)

0.001 is 1 − SLO (the budget as a ratio).

Prometheus rules¶

First, a recording rule for the error ratio at several windows (cheaper than computing in the alert):

groups:
  - name: slo:api
    rules:
      - record: job:slo_errors_ratio:rate5m
        expr: |
          sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))
      - record: job:slo_errors_ratio:rate1h
        expr: |
          sum(rate(http_requests_total{job="api",status=~"5.."}[1h]))
          / sum(rate(http_requests_total{job="api"}[1h]))
      - record: job:slo_errors_ratio:rate6h
        expr: |
          sum(rate(http_requests_total{job="api",status=~"5.."}[6h]))
          / sum(rate(http_requests_total{job="api"}[6h]))
      - record: job:slo_errors_ratio:rate3d
        expr: |
          sum(rate(http_requests_total{job="api",status=~"5.."}[3d]))
          / sum(rate(http_requests_total{job="api"}[3d]))

  - name: slo:api:alerts
    rules:
      - alert: APIErrorBudgetFastBurn
        expr: |
          job:slo_errors_ratio:rate1h > (14.4 * 0.001)
          and
          job:slo_errors_ratio:rate5m > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "API burning error budget 14.4x (whole 30d budget in ~2 days)"

      - alert: APIErrorBudgetSlowBurn
        expr: |
          job:slo_errors_ratio:rate6h > (6 * 0.001)
          and
          job:slo_errors_ratio:rate30m > (6 * 0.001)
        for: 15m
        labels: { severity: page }

      - alert: APIErrorBudgetTicket
        expr: |
          job:slo_errors_ratio:rate24h > (3 * 0.001)
          and
          job:slo_errors_ratio:rate3h > (3 * 0.001)
        for: 1h
        labels: { severity: ticket }

Latency SLOs¶

Same idea, but the "good events" are requests faster than a threshold:

1 - (
  sum(rate(http_request_duration_seconds_bucket{le="0.3",job="api"}[5m]))
  /
  sum(rate(http_request_duration_seconds_count{job="api"}[5m]))
)

That's the fraction slower than 300ms; compare it to the budget the same way.

Verification and troubleshooting¶

• No pages ever, even during a known incident — the SLI query is wrong (only counting 5xx when the outage was timeouts/499s, or the metric stopped being scraped so rate is 0/0). Test the recording rule value during a synthetic failure.
• Pages for brief spikes — you dropped the long-window condition; both windows must exceed the threshold.
• 0/0 producing NaN — no traffic. Guard with and sum(rate(http_requests_total[5m])) > 0, or accept that low-traffic services need a different (count-based) approach.
• Alert fires but budget isn't actually low — burn rate is instantaneous; add a panel showing remaining budget (1 - errors_over_30d / 0.001) so responders see the real state.
• Different services need different thresholds — the 0.001 and burn-rate multipliers are per-SLO; template them, don't hardcode one everywhere.
• Recording rules expensive — the [3d] rate over high-cardinality metrics is heavy; keep the SLI metric low-cardinality (don't break it down by every label) and let a separate query do the drill-down.

Related tools and reading¶

• On-site: Timestamp converter.
• Related posts: PromQL for beginners, Alert fatigue and how to fix it.