Loki for Logs

Introduction¶

Loki stores logs cheaply by only indexing labels (like Prometheus), not the log content. You filter by labels to narrow to a stream, then grep the content. This makes it far cheaper than Elasticsearch/OpenSearch for typical operational logs, at the cost of not being a full-text search engine.

The model¶

• A stream = a unique set of labels ({app="api", namespace="prod", level="error"}).
• Loki indexes the label set, and stores the raw log lines in chunks in object storage (S3/GCS) compressed.
• A query: {selectors} picks streams, then |= / |~ / != filter lines, then optional parsers and metric functions.

The cardinality rule (the one that matters)¶

Never put high-cardinality values in labels. A label per request_id, user_id, pod_ip, trace_id, or path creates millions of tiny streams, blows up the index, and makes Loki slow and expensive — the exact thing Loki was built to avoid.

Good labels: app, namespace, cluster, level, component, env — low cardinality, things you filter by. Everything else stays in the log line and you filter it at query time (which is fast, because you've already narrowed to a small stream).

LogQL¶

# all error lines from the api in prod
{app="api", env="prod"} |= "error"

# regex, exclude noise
{app="api"} |~ "timeout|connection refused" != "healthcheck"

# parse JSON logs and filter on a field
{app="api"} | json | status >= 500 | line_format "{{.method}} {{.path}} {{.status}}"

# parse logfmt
{app="nginx"} | logfmt | status="502"

# metrics FROM logs — error rate per minute
sum(rate({app="api", env="prod"} |= "level=error" [1m]))

# p99 of a duration extracted from the log
quantile_over_time(0.99, {app="api"} | json | unwrap duration_ms [5m])

# top paths by 5xx count
topk(10, sum by (path) (count_over_time({app="api"} | json | status>=500 [1h])))

| json / | logfmt / | pattern / | regexp extract fields at query time — so you get structured querying without indexing those fields.

Shipping logs¶

• Grafana Alloy (the current agent; supersedes Promtail) or Promtail — tails files / the journal / Docker / Kubernetes pods, attaches labels (from pod metadata, file path, static config), and pushes to Loki.
• Kubernetes — Alloy as a DaemonSet, discovering pods and labelling with namespace, pod, container, app (from pod labels). Keep the label set tight — don't label with the pod's random suffix or IP.
• Structured (JSON/logfmt) app logs make query-time parsing trivial — see Python logging configuration.

Retention and storage¶

limits_config:
  retention_period: 744h            # 31 days (global; per-tenant/stream overrides possible)
schema_config:
  configs:
    - store: tsdb
      object_store: s3
      schema: v13
compactor:
  retention_enabled: true

Storage cost ≈ compressed log volume in S3 + a small index. Way cheaper than ES. Retention is enforced by the compactor; per-stream retention rules let you keep audit logs longer than debug logs.

Loki vs Elasticsearch¶

Verification and troubleshooting¶

# In Grafana Explore, or logcli:
logcli query '{app="api"} |= "error"' --limit 100 --since 1h
logcli series '{app="api"}' --since 1h          # how many streams? (cardinality check)
logcli labels                                    # what labels exist

• Queries slow / "too many streams" / OOM — high-cardinality labels. logcli series shows the stream count for a selector; if it's thousands, you've labelled with something you shouldn't. Move it into the log line.
• maximum active stream limit exceeded / per-stream rate limit — same root cause, or a genuine traffic spike; raise max_streams_per_user / ingestion_rate_mb cautiously and fix the labels.
• Logs missing — Alloy/Promtail not tailing the file (path/permissions), the position file is stuck, timestamps parsed wrong (future-dated logs get dropped by reject_old_samples / out-of-order limits), or the push failed (Loki distributor errors).
• Can't find a log you know exists — wrong label selector (too broad = slow, too narrow = misses), time range, or the line was in a stream you didn't select. Start with {app="x"} and narrow.
• Out-of-order errors — logs arriving with older timestamps than already ingested for that stream. Newer Loki allows out-of-order within a window; ensure the agent isn't batching across a long gap.
• Retention not deleting — compactor not running / retention_enabled: false, or the object store lifecycle isn't set. Check compactor logs.

Related tools and reading¶

• On-site: Timestamp converter, Regex Tester.
• Related posts: PromQL for beginners, Prometheus cardinality management.