Worker

A Worker is a background processing component that handles tasks asynchronously from a job queue. Instead of processing work inline during a user request (which would block the response), the request handler enqueues the task and responds immediately. Workers poll the queue, process tasks independently, handle failures with retries, and use dead letter queues for unprocessable messages. This pattern is essential for video processing, email sending, payment handling, and any operation that takes longer than acceptable response time.

Idempotency means processing the same message multiple times produces the same result as processing it once. It is critical for workers because messages can be delivered more than once — if a worker crashes after processing but before acknowledging, the message is redelivered. An idempotent payment worker uses a unique transaction ID to ensure the same payment is not charged twice. Implementation strategies include unique constraint checks (idempotency keys), conditional writes (only update if current state matches expected state), and deduplication tables that record processed message IDs.

A dead letter queue captures messages that fail processing after a configured number of retries (typically 3–5). Without a DLQ, a poison message — one that always fails (invalid data, missing dependency, bug in handler code) — would be retried indefinitely, blocking the worker from processing other messages. The DLQ isolates these failures for manual investigation, automated alerting, or later reprocessing after the root cause is fixed. DLQs are a standard feature of AWS SQS, RabbitMQ, and Azure Service Bus.

Backpressure occurs when work arrives faster than workers can process it, causing unbounded queue growth. Strategies include: autoscaling workers based on queue depth (add workers when depth exceeds threshold), rate limiting producers (reject or slow down new work when the queue is too deep), flow control (producers check queue depth before enqueuing), and priority queues (process high-priority work first, deferring low-priority tasks). In Vetora, you can model queue depth growth and observe when backpressure thresholds trigger scaling or rejection.

Workers process discrete, independent tasks from a job queue — each message represents a self-contained unit of work (send an email, transcode a video, process a payment). Stream Processors handle continuous event streams with windowed aggregations, joins, and stateful transformations — analyzing patterns across many events in real time (count clicks per minute, detect fraud patterns, compute moving averages). Workers focus on task completion; stream processors focus on event analysis. Use workers for batch tasks, stream processors for continuous analytics.