PromQL for Beginners

Introduction¶

PromQL queries Prometheus's time-series data. The two things beginners get wrong: treating counters like gauges, and trying to average percentiles. Get those right and the rest follows.

Metric types¶

• Counter — only goes up (resets to 0 on restart). http_requests_total, errors_total. Never graph a counter directly — you want its rate.
• Gauge — goes up and down. memory_bytes, queue_depth, temperature_celsius. Graph directly.
• Histogram — samples bucketed into _bucket{le="..."} cumulative counts, plus _sum and _count. For latency/size distributions.
• Summary — client-side quantiles (quantile="0.99") + _sum/_count. Can't be aggregated across instances — prefer histograms.

Instant vs range vectors¶

• Instant vector — one value per series at one instant: node_load1.
• Range vector — a window of values per series: node_load1[5m]. Only useful as input to a function (rate, avg_over_time, max_over_time).

rate() and increase()¶

# per-second rate of requests, averaged over the last 5 minutes
rate(http_requests_total[5m])

# total requests in the last hour
increase(http_requests_total[1h])

# error ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m]))

• rate() handles counter resets and extrapolates to the window edges.
• The range ([5m]) should be at least 4× the scrape interval — too short and you get gaps/noise; too long and it lags. [5m] for a 15–30s scrape is typical.
• Use irate() only for fast-moving graphs where you want the last two points; it's spiky and bad for alerting.

Aggregation¶

sum(rate(http_requests_total[5m])) by (service)      # total RPS per service
avg(node_memory_MemAvailable_bytes) by (instance)
topk(5, rate(http_requests_total[5m]))                # 5 busiest series
count(up == 0) by (job)                               # how many targets down per job
sum without (instance) (rate(http_requests_total[5m]))  # sum, dropping only 'instance'

by (labels) keeps only those labels; without (labels) drops those, keeps the rest. Always aggregate rates with sum, not avg (averaging per-instance rates under-reports total throughput).

Histograms and quantiles¶

# p99 latency over 5m, aggregated across all instances
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# per route
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))

• You must sum ... by (le) before histogram_quantile — it needs the aggregated bucket counts.
• You cannot average or sum a percentile. avg(p99_latency) across instances is meaningless. Aggregate the buckets, then compute the quantile.
• Accuracy depends on bucket boundaries — if all your latency is between two buckets, the estimate is coarse. Define buckets around your SLO.

Time-shifting and comparison¶

# this week vs last week
sum(rate(http_requests_total[5m]))
  /
sum(rate(http_requests_total[5m] offset 1w))

# predict disk full
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*3600) < 0

Common mistakes¶

• Graphing http_requests_total directly — you get an ever-climbing line. Wrap in rate().
• rate(gauge[5m]) — rate is for counters; on a gauge it's nonsense. Use deriv() or avg_over_time().
• sum(rate(...)) without by — collapses everything to one line; usually you want by (service) or similar.
• Averaging percentiles — see above.
• [1m] range with a 30s scrape — only 2 samples, very noisy. [5m]+.
• Comparing _sum/_count for a "current average" — that's the average since the process started, not recent. Use rate(_sum[5m]) / rate(_count[5m]).

Verification and troubleshooting¶

• Prometheus UI → Graph tab: run the query, check the series count and values.
• "No data" — label selector typo (case-sensitive, =~ for regex), the metric name is wrong, or the target isn't being scraped (up{job="..."}).
• Values look ~2× or ~½ expected — double-counting from a relabeled/duplicated target, or a sum where you needed avg (or vice versa).
• p99 flat/stepped — bucket resolution too coarse around your latency range.
• Rate spikes to huge values then drops — counter reset (restart) not handled because the range window doesn't span it, or you used increase over a window shorter than the reset gap.
• Query slow / times out — high-cardinality series × long range. Narrow the selector, shorten the range, or use a recording rule.

Related tools and reading¶

• On-site: Timestamp converter.
• Related posts: SLO burn-rate alerts, Monitoring what actually matters.