Prometheus Cardinality Management

Introduction¶

A Prometheus time series is a unique combination of a metric name and its label values. http_requests_total{method="GET", status="200", handler="/api"} is one series. Prometheus keeps the head of every active series in memory, so series count drives RAM. High cardinality is the #1 way to break a Prometheus.

How labels multiply¶

Cardinality is roughly the product of each label's distinct values:

http_requests_total{method, status, handler}
  method:  5 values
  status:  8 values
  handler: 40 values
  -> 5 × 8 × 40 = 1,600 series   (per target — multiply by scrape targets)

Add user_id (100,000 values) → 160 million series. Add path with IDs in it (/orders/48213) → unbounded. Each of these OOMs Prometheus.

Never label with: user/customer/tenant IDs, request IDs, trace IDs, session IDs, email addresses, full URLs/paths with IDs, timestamps, pod IPs, full version strings, error messages, or anything unbounded.

Fine to label with: method, status code (or status class), a templated route (/orders/:id), instance/job, namespace, a small enum.

Finding the offenders¶

# The TSDB status page (Prometheus UI → Status → TSDB Status):
#   - top 10 metrics by series count
#   - top 10 label names by distinct value count
#   - top 10 label VALUES by series count

# Queries:
topk(10, count by (__name__)({__name__=~".+"}))              # series per metric
count(count by (le) (http_request_duration_seconds_bucket))   # histogram bucket count
prometheus_tsdb_head_series                                    # total active series
scrape_samples_scraped                                         # per-target sample count — a spiking target

# promtool
promtool tsdb analyze /path/to/data

The scrape_samples_scraped for a target suddenly jumping = an app started emitting a high-cardinality metric. Alert on it: scrape_samples_scraped > 10000.

Fixing it¶

Drop labels or metrics at scrape time¶

scrape_configs:
  - job_name: myapp
    metric_relabel_configs:
      # drop a label entirely
      - regex: "user_id|request_id|trace_id"
        action: labeldrop
      # drop a whole noisy metric
      - source_labels: [__name__]
        regex: "myapp_internal_debug_.*"
        action: drop
      # keep only specific metrics (allowlist — aggressive)
      - source_labels: [__name__]
        regex: "myapp_(http_requests_total|http_request_duration_seconds_.*|up)"
        action: keep

metric_relabel_configs runs after scrape, before storage — the app still emits it, you just don't keep it.

Fix the app¶

The real fix: the app shouldn't emit path="/orders/48213" — it should emit route="/orders/:id" (the matched route template). Most instrumentation libraries do this if configured; a hand-rolled counter with the raw path is the bug.

Reduce histogram buckets¶

Each histogram is (buckets + 2) series per label combo. A 15-bucket histogram across method × status × route is a lot. Trim buckets to the ones around your SLO, or use native histograms (Prometheus 2.40+) which are far more efficient.

Recording rules for expensive queries¶

If a dashboard query aggregates millions of series every 15s, pre-compute it:

groups:
  - name: aggregations
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job, status) (rate(http_requests_total[5m]))

Doesn't reduce ingestion cardinality, but reduces query cost and downstream (remote-write) cardinality if you only ship the recorded series.

The memory cost¶

Rule of thumb: ~1–3 KB of RAM per active series for the head block, plus query working memory. 1M series ≈ 2–4 GB just for the head. 10M series needs a big box and careful tuning (--storage.tsdb.retention, chunk settings), or sharding Prometheus, or a scalable backend (Mimir, Thanos, Cortex, VictoriaMetrics).

Verification and troubleshooting¶

prometheus_tsdb_head_series
rate(prometheus_tsdb_head_series_created_total[5m])   # churn — new series appearing
prometheus_tsdb_head_chunks
process_resident_memory_bytes{job="prometheus"}

• Prometheus OOM-killed / restart loop — too many series for its RAM. TSDB Status page for the top metrics/labels; labeldrop/drop the worst, fix the app, then let it recover (it replays the WAL — give it headroom).
• prometheus_tsdb_head_series_created_total climbing steadily — churn: labels with values that constantly change (a new pod name every deploy is normal-ish; a new label value per request is a bug). Even if the active count looks OK, churn bloats the index and slows compaction.
• Queries timing out — a query touching millions of series. Add a recording rule, narrow the selector, or reduce cardinality at the source.
• Remote-write lagging / expensive — you're shipping high-cardinality series to Mimir/Grafana Cloud and paying per series. write_relabel_configs to drop before sending.
• One target dwarfs the rest — scrape_samples_scraped by job/instance. That app added a bad metric; talk to its owners, metric_relabel_configs as a stopgap.
• Histogram queries slow — too many buckets × labels. Native histograms, or fewer buckets, or aggregate by (le) in a recording rule.

Related tools and reading¶

• On-site: (no dedicated tool) — see Regex Tester for relabel regexes.
• Related posts: PromQL for beginners, A node_exporter metrics guide.