The Prometheus Blackbox Exporter

Introduction¶

White-box metrics come from inside your app. The blackbox exporter probes from outside: it makes an HTTP request, opens a TCP connection, pings a host, or resolves a name, and reports success/failure and timings. It catches "the whole thing is down", DNS breakage, expired certs, and redirect loops that internal metrics miss.

Modules¶

blackbox.yml defines named probe modules:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []          # defaults to 2xx
      method: GET
      fail_if_ssl: false
      fail_if_not_ssl: true           # require HTTPS
      preferred_ip_protocol: ip4

  http_post_json:
    prober: http
    http:
      method: POST
      headers: { Content-Type: application/json }
      body: '{"ping":true}'
      valid_status_codes: [200, 201]

  tcp_connect:
    prober: tcp
    timeout: 5s

  icmp:
    prober: icmp
    timeout: 5s
    icmp: { preferred_ip_protocol: ip4 }

  dns_soa:
    prober: dns
    dns:
      query_name: example.com
      query_type: SOA
      validate_answer_rrs:
        fail_if_not_matches_regexp: [".*"]

The multi-target pattern¶

You run one blackbox exporter and point Prometheus at it once per target via relabeling. Prometheus scrapes blackbox:9115/probe?target=<url>&module=<name>:

scrape_configs:
  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com
          - https://api.example.com/health
          - https://blog.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115        # the exporter's real address

The relabel_configs block is the magic: it moves the target into the target URL param and rewrites __address__ to the exporter. The result: each URL becomes a distinct scrape with instance="https://example.com".

Metrics you get¶

probe_success                          1 or 0
probe_duration_seconds
probe_http_status_code
probe_http_duration_seconds{phase="resolve|connect|tls|processing|transfer"}
probe_ssl_earliest_cert_expiry         unix timestamp
probe_http_redirects
probe_dns_lookup_time_seconds

Useful alerts¶

- alert: EndpointDown
  expr: probe_success == 0
  for: 2m
  labels: { severity: page }
  annotations:
    summary: "{{ $labels.instance }} probe failing"

- alert: CertExpiringSoon
  expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
  for: 1h
  labels: { severity: ticket }
  annotations:
    summary: "{{ $labels.instance }} TLS cert expires in {{ $value | printf \"%.0f\" }} days"

- alert: ProbeSlow
  expr: probe_http_duration_seconds{phase="processing"} > 2
  for: 10m
  labels: { severity: ticket }

The cert-expiry alert alone justifies deploying the exporter — one place to catch every forgotten renewal.

Where to run it¶

• Inside the cluster/VPC — checks "can our stuff reach our stuff", catches internal breakage.
• From an external location (a small box in another region/cloud) — the true user perspective; catches DNS, CDN, edge LB, and full-region failures.
• Ideally both, with a label distinguishing the vantage point.

Verification and troubleshooting¶

# Probe by hand
curl -s 'http://blackbox:9115/probe?target=https://example.com&module=http_2xx&debug=true'

debug=true returns the full probe log — DNS resolution, TLS handshake, HTTP exchange — invaluable for "why is probe_success 0".

• probe_success 0 but the site is up — the module expects something it isn't getting: fail_if_not_ssl on an HTTP URL, valid_status_codes not including a 301, fail_if_body_not_matches_regexp too strict, or IPv6 preference hitting a broken AAAA. debug=true shows which check failed.
• ICMP probes all fail — the exporter needs CAP_NET_RAW (container: --cap-add=NET_RAW; or run as root). Also many hosts/clouds block ICMP.
• No metrics for a target — relabeling wrong; up{job="blackbox-http"} and the Prometheus targets page. __param_target must be set from __address__.
• All targets share one instance label — the relabel step setting instance from __param_target is missing.
• TLS cert metric absent — probe didn't reach TLS (connection refused, DNS failure), or it's an HTTP (not HTTPS) target.
• Probe duration includes DNS every time — the exporter doesn't cache DNS by design (it's testing resolution too). That's a feature.

Related tools and reading¶

• On-site: SSL Certificate Checker, DNS lookup, HTTP Header Analyzer, Monitors / uptime.
• Related posts: Monitoring what actually matters, PromQL for beginners.