Every Node.js API eventually hits the same wall: a request that takes 8 seconds because it is sending an email, rendering a PDF invoice and resizing three images before returning a response. The fix is not a bigger server. The fix is a queue.

This guide is a practical, production-oriented tutorial on using BullMQ with Node.js and Redis. No toy examples that break the first time a worker restarts mid-job. We cover queue configuration, worker setup, retries with backoff, concurrency limits, rate limiting, monitoring failed jobs and graceful shutdown, which is the part most tutorials forget and the part that costs you data in production.

Why background jobs matter in Node.js

Node.js runs your JavaScript on a single thread. Anything slow or CPU heavy that you run inside a request handler has two consequences:

• The client waits. Your p95 latency explodes and mobile clients time out.
• The event loop is blocked. CPU bound work such as image resizing or PDF rendering starves every other pending request on that process.

A queue solves both. The API does the minimum work needed to be correct (write to the database, push a job), returns 202 Accepted in a few milliseconds, and a separate worker process does the heavy lifting. If the worker crashes, the job is still in Redis and gets picked up again.

What is BullMQ in Node.js?

BullMQ is a Redis based distributed job queue library for Node.js, written in TypeScript and shipped with native typings. It is the successor of Bull (the older OptimalBits/bull package) and is built on modern Redis data structures, with Lua scripts guaranteeing atomic state transitions.

What you get out of the box:

• Job persistence in Redis, so jobs survive a process restart
• Automatic retries with configurable backoff
• Delayed jobs and cron style job schedulers
• Per worker concurrency and per queue rate limiting
• Priorities, deduplication by job id, parent/child flows
• Stalled job detection when a worker dies without finishing
• Events you can stream to a dashboard or your observability stack

Prerequisites

• Node.js 20 LTS or newer (22 LTS recommended)
• Redis 6.2 or newer, or a compatible service such as Valkey, AWS ElastiCache or Upstash
• BullMQ v5 (the API shown below uses the v5 conventions)

npm install bullmq ioredis

Step 1: a single, reusable Redis connection

The most common production bug with BullMQ is an incorrectly configured Redis client. Workers use blocking commands, so maxRetriesPerRequest must be null. Create one connection module and import it everywhere.

// src/redis.js
import IORedis from 'ioredis';

export const connection = new IORedis(process.env.REDIS_URL, {
  maxRetriesPerRequest: null,
  enableReadyCheck: false,
  // TLS for managed Redis providers
  // tls: {},
});

connection.on('error', (err) => {
  console.error('[redis] connection error', err.message);
});

Tip: use a dedicated Redis database or a prefix option per environment so that staging jobs never leak into production queues. For the wider picture, see BullMQ with AppSignal for Node.js.

Step 2: declare your queues with sane defaults

Do not set job options at every call site. Put them in defaultJobOptions so every producer inherits the same retry policy and cleanup rules.

// src/queues.js
import { Queue } from 'bullmq';
import { connection } from './redis.js';

const defaultJobOptions = {
  attempts: 5,
  backoff: { type: 'exponential', delay: 2000 },
  removeOnComplete: { age: 3600, count: 1000 },
  removeOnFail: { age: 7 * 24 * 3600 },
};

export const emailQueue = new Queue('email', { connection, defaultJobOptions });
export const pdfQueue   = new Queue('pdf',   { connection, defaultJobOptions });
export const imageQueue = new Queue('image', { connection, defaultJobOptions });

Why removeOnComplete and removeOnFail matter: without them, Redis memory grows forever. Keeping the last 1000 completed jobs for one hour is usually enough for debugging, while failed jobs are worth keeping for a week so you can inspect and replay them.

The job options you will actually use

Option	What it does	Typical value
attempts	Total number of tries before the job is marked failed	3 to 5
backoff	Delay strategy between retries (fixed, exponential or custom)	exponential, 2000 ms
delay	Wait before the job becomes available	0 or a few seconds
priority	Lower number is processed first	1 for transactional, 10 for batch
jobId	Custom id used for deduplication	invoice-pdf-8412
removeOnComplete	Auto cleanup of successful jobs	{ age: 3600, count: 1000 }

Step 3: push jobs from your API without blocking the request

The API route stays fast. It persists the business entity, enqueues the work and answers immediately.

// src/routes/invoices.js
import express from 'express';
import { pdfQueue, emailQueue } from '../queues.js';

const router = express.Router();

router.post('/invoices', async (req, res, next) => {
  try {
    const invoice = await db.invoices.create(req.body);

    // jobId makes the enqueue idempotent: a double click will not
    // generate the same PDF twice while the first job is still around
    await pdfQueue.add(
      'generate-invoice-pdf',
      { invoiceId: invoice.id },
      { jobId: `invoice-pdf-${invoice.id}`, priority: 1 }
    );

    await emailQueue.add('invoice-created', {
      to: invoice.customerEmail,
      template: 'invoice-created',
      invoiceId: invoice.id,
    });

    res.status(202).json({ id: invoice.id, status: 'processing' });
  } catch (err) {
    next(err);
  }
});

export default router;

Golden rule: never put large payloads in the job data. Store the file, the HTML or the image in S3 or in your database and pass an id or a key. Job data lives in Redis memory and is serialized as JSON on every state change.

Step 4: the worker process

Workers should run as a separate process (a separate container or a separate npm run worker entry point), not inside your web server. That way you scale API and processing independently and a CPU heavy job never slows an HTTP response.

// src/workers/email.worker.js
import { Worker, UnrecoverableError } from 'bullmq';
import { connection } from '../redis.js';
import { sendEmail } from '../services/mailer.js';

const worker = new Worker(
  'email',
  async (job) => {
    const { to, template, invoiceId } = job.data;

    if (!to || !to.includes('@')) {
      // A bad address will never succeed, so do not waste 5 retries on it
      throw new UnrecoverableError(`Invalid recipient: ${to}`);
    }

    await job.updateProgress(10);
    const result = await sendEmail({ to, template, invoiceId });
    await job.updateProgress(100);

    return { messageId: result.messageId };
  },
  {
    connection,
    concurrency: 10,
    limiter: { max: 100, duration: 60_000 }, // provider rate limit
  }
);

worker.on('completed', (job, result) => {
  console.log(`[email] job ${job.id} done`, result.messageId);
});

worker.on('failed', (job, err) => {
  console.error(`[email] job ${job?.id} failed (attempt ${job?.attemptsMade})`, err.message);
});

worker.on('error', (err) => {
  console.error('[email] worker error', err);
});

Concurrency versus rate limiting

• concurrency is how many jobs a single worker instance processes in parallel. For I/O bound work (HTTP calls, SMTP, database) 10 to 50 is fine. For CPU bound work such as Sharp image resizing or Puppeteer PDF rendering, keep it at 1 to 4 and scale horizontally instead.
• limiter is a shared rate limit across all workers of the queue. Use it to respect third party API quotas, for example 100 emails per minute.

A useful pattern is one queue per workload profile:

Queue	Workload	Concurrency	Notes
email	I/O bound, fast	20	Add a limiter matching your ESP quota
pdf	CPU + memory heavy	2	Reuse a browser pool, raise lockDuration
image	CPU bound	number of cores	Use a sandboxed processor to keep the event loop free
webhooks	Unreliable third parties	15	High attempts, exponential backoff

Keep the event loop free with a sandboxed processor

For CPU heavy jobs, pass a file path instead of a function. BullMQ then runs the processor in a child process, so a 4 second image transform cannot block the worker heartbeat.

// src/workers/image.worker.js
import { Worker } from 'bullmq';
import path from 'node:path';
import { connection } from '../redis.js';

const processorFile = path.join(process.cwd(), 'src/processors/resize.js');

export const imageWorker = new Worker('image', processorFile, {
  connection,
  concurrency: 4,
  lockDuration: 120_000, // long jobs need a longer lock
});

// src/processors/resize.js
import sharp from 'sharp';
import { getObject, putObject } from '../services/s3.js';

export default async function (job) {
  const { key, widths } = job.data;
  const original = await getObject(key);

  for (const width of widths) {
    const buffer = await sharp(original)
      .resize({ width })
      .webp({ quality: 82 })
      .toBuffer();

    await putObject(`${key}-${width}.webp`, buffer);
    await job.updateProgress(Math.round((widths.indexOf(width) + 1) / widths.length * 100));
  }

  return { variants: widths.length };
}

Step 5: retries, backoff and poison jobs

Retries are what make a queue reliable, but blind retries create duplicates and infinite loops. Three rules:

1. Make processors idempotent. Before sending an email or charging a card, check whether the side effect already happened. Store a marker keyed by the job id or the business entity.
2. Use exponential backoff. A third party API that is down at 2000 ms will usually still be down at 2100 ms. Exponential backoff gives it room to recover.
3. Fail fast on permanent errors. A 400 from an API or a malformed payload will never succeed, so throw UnrecoverableError and stop burning attempts.

You can also define a custom backoff strategy, for example to honour a Retry-After header:

const worker = new Worker('webhooks', processor, {
  connection,
  settings: {
    backoffStrategy: (attemptsMade, type, err) => {
      if (err?.retryAfterMs) return err.retryAfterMs;
      return Math.min(2 ** attemptsMade * 1000, 5 * 60_000); // cap at 5 min
    },
  },
});

Step 6: graceful shutdown (the part everyone skips)

When Kubernetes, ECS or your PaaS deploys a new version, it sends SIGTERM and then kills the process after a grace period. If you do nothing, a job that was 90 percent done is killed in the middle, and BullMQ will only recover it after the stalled check, which means duplicated or delayed work.

// src/workers/index.js
import { emailWorker } from './email.worker.js';
import { pdfWorker } from './pdf.worker.js';
import { imageWorker } from './image.worker.js';
import { connection } from '../redis.js';

const workers = [emailWorker, pdfWorker, imageWorker];
let shuttingDown = false;

async function shutdown(signal) {
  if (shuttingDown) return;
  shuttingDown = true;
  console.log(`[worker] ${signal} received, draining jobs...`);

  try {
    // close() stops fetching new jobs and waits for active ones to finish
    await Promise.all(workers.map((w) => w.close()));
    await connection.quit();
    console.log('[worker] clean shutdown');
    process.exit(0);
  } catch (err) {
    console.error('[worker] shutdown error', err);
    process.exit(1);
  }
}

for (const signal of ['SIGTERM', 'SIGINT']) {
  process.on(signal, () => shutdown(signal));
}

// Safety net: never hang forever
process.on('SIGTERM', () => {
  setTimeout(() => process.exit(1), 30_000).unref();
});

Two things to align: your orchestrator grace period (terminationGracePeriodSeconds in Kubernetes) must be longer than your longest expected job, and lockDuration must be longer than your longest job execution time, otherwise the job is considered stalled while it is still running.

Step 7: monitoring failed jobs

A queue you cannot observe is a queue you do not trust. Three complementary layers work well.

1. Queue events for metrics and alerting

import { QueueEvents } from 'bullmq';
import { connection } from './redis.js';

const events = new QueueEvents('email', { connection: connection.duplicate() });

events.on('failed', ({ jobId, failedReason }) => {
  metrics.increment('queue.email.failed');
  logger.error({ jobId, failedReason }, 'email job failed');
});

events.on('stalled', ({ jobId }) => {
  metrics.increment('queue.email.stalled');
});

2. A health endpoint with queue depth

app.get('/health/queues', async (req, res) => {
  const counts = await emailQueue.getJobCounts(
    'waiting', 'active', 'delayed', 'failed', 'completed'
  );
  const healthy = counts.failed < 50 && counts.waiting < 5000;
  res.status(healthy ? 200 : 503).json(counts);
});

Alert on two signals in particular: growing waiting count (you are producing faster than you consume, add workers) and growing failed count (something is broken downstream).

3. Inspecting and replaying failed jobs

// scripts/retry-failed.js
import { emailQueue } from '../src/queues.js';

const failed = await emailQueue.getFailed(0, 100);

for (const job of failed) {
  console.log(job.id, job.name, job.attemptsMade, job.failedReason);
  // Replay only what is safe to replay
  if (!job.failedReason.includes('Invalid recipient')) {
    await job.retry();
  }
}

await emailQueue.close();
process.exit(0);

For a UI, @bull-board/express or bull-board gives you a dashboard of waiting, active, delayed and failed jobs with one click retry. Mount it behind authentication, never expose it publicly, because job data often contains personal information.

import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter.js';
import { ExpressAdapter } from '@bull-board/express';

const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath('/admin/queues');

createBullBoard({
  queues: [new BullMQAdapter(emailQueue), new BullMQAdapter(pdfQueue)],
  serverAdapter,
});

app.use('/admin/queues', requireAdmin, serverAdapter.getRouter());

Step 8: recurring jobs with Job Schedulers

In BullMQ v5, repeatable jobs are handled by Job Schedulers, which replace the older repeat option. A scheduler is upserted by key, so redeploying does not create duplicates. gopenai.com has a solid rundown on this.

import { reportQueue } from './queues.js';

await reportQueue.upsertJobScheduler(
  'daily-sales-report',
  { pattern: '0 3 * * *', tz: 'Europe/Paris' },
  {
    name: 'sales-report',
    data: { scope: 'daily' },
    opts: { attempts: 3, removeOnComplete: true },
  }
);

// Interval based alternative
await reportQueue.upsertJobScheduler('sync-crm', { every: 15 * 60_000 });

This replaces a cron container calling an HTTP endpoint, and you keep retries, history and observability for free.

Chaining work with Flows

Some pipelines need ordering: generate a PDF, then upload it, then send the email with the link. BullMQ Flows let a parent job wait for its children.

import { FlowProducer } from 'bullmq';
import { connection } from './redis.js';

const flow = new FlowProducer({ connection });

await flow.add({
  name: 'notify-customer',
  queueName: 'email',
  data: { invoiceId: 8412 },
  children: [
    { name: 'render-pdf', queueName: 'pdf', data: { invoiceId: 8412 } },
    { name: 'upload-pdf', queueName: 'storage', data: { invoiceId: 8412 } },
  ],
});

Inside the parent processor, await job.getChildrenValues() gives you the results returned by the children.

BullMQ compared to other options

Tool	Best for	Trade off
BullMQ	Node.js background jobs, retries, scheduling, per job state	Requires Redis, memory bound, not built for millions of events per second
RabbitMQ	Complex routing between many services and languages	More infrastructure, no first class job scheduling or progress tracking
Kafka	High throughput event streaming and replayable logs	Not a task queue, no per job retry or delay semantics out of the box
node-cron in process	A single tiny periodic task	Runs on every instance, no retries, no persistence, blocks the event loop
SQS or Cloud Tasks	Fully managed, serverless friendly	Per request cost, higher latency, fewer local dev conveniences

Production checklist

1. Workers run in a separate process from the HTTP API
2. maxRetriesPerRequest: null on the worker Redis connection
3. defaultJobOptions with attempts, backoff, removeOnComplete and removeOnFail
4. Job payloads contain ids, not blobs
5. Processors are idempotent, permanent errors throw UnrecoverableError
6. lockDuration greater than your longest job, and matching orchestrator grace period
7. SIGTERM and SIGINT handlers calling worker.close()
8. Alerts on waiting depth, failed count and stalled events
9. Redis persistence enabled (AOF) and maxmemory-policy set to noeviction for the queue instance
10. A protected dashboard and a replay script for failed jobs

Important: never point BullMQ at a Redis instance configured as an LRU cache. If Redis evicts keys, jobs disappear silently. Use a dedicated instance for queues.

Common mistakes we see in audits

• Creating a new Queue instance inside a request handler. Every instance opens Redis connections. Create them once at module level.
• Forgetting to await queue.add(). An unhandled rejection means the job is silently lost.
• Very high concurrency on CPU bound jobs. Twenty parallel Puppeteer renders will not go faster, they will exhaust memory.
• No cleanup. Millions of completed jobs in Redis, then an out of memory incident at 3 a.m.
• Storing secrets in job data. Job data is readable by anyone with Redis or dashboard access.

FAQ

What is BullMQ in Node.js?

BullMQ is a Redis based message and job queue library for Node.js. It lets you move slow work such as email sending, PDF generation and image processing out of the HTTP request cycle into background workers, with retries, delays, priorities, concurrency control and scheduling built in.

Is BullMQ similar to Kafka?

No. Kafka is a distributed event log designed for very high throughput streaming and replay. BullMQ is a task queue: each job has a lifecycle (waiting, active, completed, failed), individual retries, delays and progress. Use Kafka for event streams between services, BullMQ for background jobs inside a Node.js application.

Are RabbitMQ and BullMQ the same?

They solve overlapping problems with different designs. RabbitMQ is a standalone broker with advanced routing (exchanges, bindings) and is language agnostic. BullMQ is a library that uses Redis as storage, is focused on Node.js and gives you job level features such as backoff strategies, repeatable schedulers and flows with far less operational overhead.

Is BullMQ available on npm?

Yes, install it with npm install bullmq. It ships TypeScript definitions, and clients also exist for Python and other runtimes so non Node services can produce jobs for the same Redis queues.

Should I use Bull or BullMQ?

Use BullMQ for any new project. The original Bull package is in maintenance mode, while BullMQ gets the new features (flows, job schedulers, telemetry hooks) and native TypeScript support.

How many workers should I run?

Start with one worker process per queue profile and scale horizontally based on the waiting count. For I/O bound queues, raise concurrency first. For CPU bound queues, keep concurrency close to the number of available cores and add more worker containers.

Does BullMQ guarantee exactly once processing?

No queue really does. BullMQ gives at least once delivery: if a worker dies mid job, the job is recovered and retried. Your processors must be idempotent so a retry does not send the same email twice.

Need help putting this in production?

Designing queue topology, sizing workers, setting lock durations and wiring alerting is where most Node.js projects lose time. At Box Software we build and audit production Node.js backends, including BullMQ and Redis based job pipelines. If your API is slow because it does too much inside the request, get in touch and we will review your architecture.