Redis Streams as a Work Queue

Introduction¶

Redis Streams (XADD and friends, since Redis 5) are an append-only log with consumer groups — multiple workers share the load, each message is delivered to one worker in the group, and un-acked messages can be redelivered. It's a capable lightweight queue when you already run Redis and don't want to operate Kafka/RabbitMQ.

Producing¶

XADD jobs '*' type resize path /uploads/a.jpg
# '*' = auto-generate the ID (timestamp-seq). Returns e.g. 1724764800000-0

Cap the stream so it doesn't grow forever:

XADD jobs MAXLEN '~' 100000 '*' type resize path /uploads/a.jpg
#   '~' = approximate trim (much cheaper); trims to ~100k entries
# or by age (Redis 6.2+):
XADD jobs MINID '~' <id-from-1h-ago> '*' ...

Consumer groups¶

XGROUP CREATE jobs workers '$' MKSTREAM      # start from new messages; MKSTREAM if the stream doesn't exist
# ('0' instead of '$' to consume the whole history)

Each worker reads with a unique consumer name:

XREADGROUP GROUP workers worker-1 COUNT 10 BLOCK 5000 STREAMS jobs '>'
#   '>'  = "give me messages never delivered to this group"
# process the messages, then:
XACK jobs workers 1724764800000-0 1724764800001-0

XREADGROUP with > delivers each message to exactly one consumer and records it in the group's Pending Entries List (PEL) until XACKed.

Handling failures — the PEL and claiming¶

If a worker dies before XACK, the message stays in the PEL. Recover it:

# what's pending, and for how long?
XPENDING jobs workers
XPENDING jobs workers - + 100 IDLE 60000       # entries idle > 60s

# claim idle messages to a live worker (Redis 6.2+)
XAUTOCLAIM jobs workers worker-2 60000 0 COUNT 10
#   reassigns messages idle > 60s to worker-2; process + XACK them

Run an XAUTOCLAIM sweep periodically (a janitor loop) so a crashed worker's messages get retried. Track a delivery count (XPENDING shows it) and move messages that fail repeatedly to a dead-letter stream (XADD jobs:dead ... then XACK + XDEL the original).

Semantics¶

• At-least-once. A message can be delivered more than once (worker processed it, then died before XACK; the janitor reclaims and redelivers). Consumers must be idempotent.
• Ordering is per-stream (IDs are monotonic). Within a consumer group, parallelism breaks strict ordering — like SQS standard.
• Persistence is Redis's persistence (RDB/AOF — see RDB vs AOF). appendfsync everysec = up to ~1s of acked-but-not-persisted messages lost on a crash. For stricter durability, appendfsync always (slow) or replicate + WAIT.

When to use a real broker instead¶

Streams are enough for: background jobs, event fan-in, moderate throughput, "I already run Redis". Choose Kafka/RabbitMQ/NATS when you need:

• Very high throughput / long retention / replay at scale (Kafka).
• Complex routing / topic exchanges / priorities / delayed messages (RabbitMQ).
• Exactly-once-ish processing with transactions (Kafka + care).
• Multi-datacenter replication of the queue itself.
• Durability guarantees stronger than Redis persistence gives you.

Verification and troubleshooting¶

XINFO STREAM jobs
XINFO GROUPS jobs
XINFO CONSUMERS jobs workers
XLEN jobs
XPENDING jobs workers

• Messages produced but consumers get nothing — group created with '$' (only new messages) after the messages were added; consuming with '>' only gets undelivered ones. Recreate the group at '0', or XADD new messages.
• Same message processed twice — at-least-once; a worker didn't XACK before dying, the janitor reclaimed it. Make the handler idempotent (dedupe on the message ID or a business key).
• PEL growing without bound — workers process but never XACK (bug), or the janitor isn't running. XPENDING shows the backlog per consumer.
• Memory growing — no MAXLEN/MINID trim on XADD, so the stream keeps every message forever. Trim on produce, or run XTRIM periodically. Note: trimming removes entries even if unacked — don't trim below your retry window.
• XAUTOCLAIM not finding stalled messages — the min-idle-time is longer than the messages have been idle, or you're claiming from the wrong start ID.
• Consumer names accumulating (XINFO CONSUMERS shows dozens of dead ones) — each restart with a new random name leaves a stale consumer. Use stable names (pod name / worker id), and XGROUP DELCONSUMER for departed ones.
• Throughput ceiling — single-threaded Redis; one stream is limited by one core's command rate. Shard across multiple streams (jobs:0..N) with a consumer per shard if you hit it.

Related tools and reading¶

• On-site: Timestamp converter.
• Related posts: SQS vs SNS vs EventBridge, MQTT QoS levels explained.