Deduplication

Source URL: https://docs.bullmq.io/patterns/deduplication

Deduplication

Sometimes, you may want to decide when you want to stop deduplicating jobs.

Until job is active

As soon as job is moved to active, you must call removeDeduplicationKey method:

import { Job, Queue, Worker } from 'bullmq';

const myQueue = new Queue('Paint');

const worker = new Worker('Paint', async (job: Job) => {
  await job.removeDeduplicationKey();
  console.log('Do something with job');
  return 'some value';
});

myQueue.add('house', { color: 'white' }, { deduplication: { id: 'house' } });

{% hint style=“info” %} Previous example uses Simple Mode but it can be combined with Throttle Mode or Debounce Mode. {% endhint %}

Using job schedulers

Sometimes it is desired to deduplicate jobs that are generated by job schedulers to save resources and avoid unnecessary work:

Deduplication options are not available in JobSchedulerTemplateOptions because they could interfere with job creation from scheduler templates. Since a new job is added as soon as the previous one moves to the active state, deduplication could disrupt this process by preventing the addition of this new record. However, there is an alternative to handle this. Let’s look at an example:

import { Queue, Worker } from 'bullmq';

const myQueue = new Queue('Paint');

const worker = new Worker(
  'Paint',
  async job => {
    if (job.name === 'paint-trigger') {
      // Add a job that will be deduplicated for 90 seconds.
      await myQueue.add(
        'house',
        { color: 'white' },
        { deduplication: { id: 'customValue', ttl: 90000 } },
      );
    }
  },
  { connection },
);

await myQueue.upsertJobScheduler('repeat', {
  pattern: '* * * * *', // every minute
  template: {
    name: 'paint-trigger',
    data: {},
  },
});

In this way, you can deduplicate a job when using job schedulers.

Read more:

• 💡 Add Job API Reference
• 💡 Deduplication Reference
• 💡 Remove Deduplication Key API Reference
• 💡 Upsert Job Scheduler API Reference