OpenTelemetry for Distributed Tracing

Introduction¶

Metrics tell you something is slow; logs tell you what one service did. Distributed tracing follows a single request through every service it touches, showing where the time went. OpenTelemetry (OTel) is the CNCF-standard, vendor-neutral instrumentation.

The model¶

• Span — one unit of work: an HTTP handler, a DB query, an RPC. Has a name, start/end time, attributes (http.method, db.statement), status, and events.
• Trace — a tree of spans sharing a trace ID, representing one end-to-end request. Each span has a parent span ID.
• Context propagation — the trace ID + parent span ID travel between services in a header (W3C traceparent, the standard; older: B3). Every hop must read the incoming context and inject it into outgoing calls, or the trace breaks into disconnected pieces.

Getting spans¶

Three levels, combine them:

1. Auto-instrumentation — a language agent that wraps common libraries (HTTP servers/clients, DB drivers, gRPC, messaging) with zero code changes. Java -javaagent, Python opentelemetry-instrument, Node --require @opentelemetry/auto-instrumentations-node/register. Start here.
2. Manual spans — wrap the business logic that matters:

   from opentelemetry import trace
   tracer = trace.get_tracer(__name__)
   
   with tracer.start_as_current_span("price_cart") as span:
       span.set_attribute("cart.items", len(items))
       total = compute(items)
       span.set_attribute("cart.total", total)
   

3. Framework/library instrumentation packages — pre-built for FastAPI, Django, SQLAlchemy, Kafka, etc.

The Collector¶

Don't export straight from every app to your backend. Run the OpenTelemetry Collector (as a sidecar, DaemonSet, or gateway):

receivers:
  otlp:
    protocols: { grpc: {}, http: {} }
processors:
  batch: {}
  memory_limiter: { check_interval: 1s, limit_mib: 400 }
  resource:
    attributes:
      - { key: deployment.environment, value: prod, action: upsert }
exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: true }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/tempo]

The Collector gives you: one place to change the backend, batching/retry, scrubbing PII from attributes, tail-based sampling, and protocol translation.

Sampling — or you'll go broke¶

Tracing every request at scale is expensive to store and query.

• Head sampling — decide at the start of the trace (e.g. keep 5%). Simple, but you might drop the 1 trace that had the error.
• Tail sampling — the Collector buffers a whole trace, then decides: keep all errors, all slow traces, and a small % of the rest. Much better signal; needs the tailsamplingprocessor and enough Collector memory to buffer.

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow, type: latency, latency: { threshold_ms: 500 } }
      - { name: sample-rest, type: probabilistic, probabilistic: { sampling_percentage: 5 } }

Reading a trace¶

In Jaeger/Tempo/Grafana, a trace is a waterfall:

• Wide gaps between spans = time not attributed to any span — usually network, a queue wait, or missing instrumentation.
• One span dominates = there's your bottleneck. Drill into its attributes (the SQL, the downstream URL).
• Many identical sibling spans = an N+1 (a loop making one query per item).
• A span with an error status propagates up — the failing leaf is the root cause.

Verification and troubleshooting¶

# Is the app exporting? Point it at a debug collector:
#   exporters: { debug: { verbosity: detailed } }
otelcol validate --config collector.yaml
curl -v http://collector:13133/           # Collector health check

• Trace shows only one service (context not propagated) — the outgoing HTTP client isn't instrumented, or a proxy/gateway strips the traceparent header, or two services use different propagators (W3C vs B3). Standardise on W3C.
• No spans at all — SDK not initialised before the app starts, wrong OTLP endpoint/port (4317 gRPC, 4318 HTTP), or TLS mismatch. Use the debug exporter to confirm spans are produced.
• Traces truncated / missing spans — sampling dropped them, or the span wasn't ended (with block / span.end()), or the process exited before the batch flushed (add a shutdown hook / force_flush).
• Collector OOM — tail sampling buffering too many traces; raise memory, lower decision_wait, or move to head sampling.
• Attributes contain PII / huge SQL blobs — scrub in the Collector (attributes processor delete/hash), and don't put full request bodies in attributes.
• Clock skew makes spans look negative — child span "starts before" parent. Fix NTP on the hosts.

Related tools and reading¶

• On-site: Timestamp converter, JSON formatter.
• Related posts: Monitoring what actually matters, PromQL for beginners.