[question] Concurrency when using 1 second interval

[Jan Van Echelpoel]

In my app I need to run a job each second to see if the finished_at value of some records has passed, and if so, delete it. In order to do so I set up a job with interval: '1s' where a connection with MongoDB is made (through Mongoose) and the action is performed.

The problem is, because the job takes longer than 1 second to complete, I'm getting lots of Error: Job is already running errors because the next job is already starting. I believe the solution for this is implementing concurrency (through p-queue as stated in the docs) but I can't get it to work.

Any ideas to point me in the right direction?
Thanks!

[shadowgate15]

I would probably create one long running task and then use p-queue to actually manage the removal of the docs.

Another option would be to setup multiple jobs firing at different intervals. Although this could lead to a race condition.

[Jan Van Echelpoel]

Thanks, got it working with a long running task:

index.ts

const bree = new Bree({
  root: path.join(__dirname, 'src/jobs'),
  defaultExtension: process.env.NODE_ENV === 'development' ? 'ts' : 'js',
  jobs: [
    { name: 'check-finished' }
  ]
})
const graceful = new Graceful({ brees: [bree] })

graceful.listen()
bree.start()

src/jobs/check-finished.ts

import { parentPort } from 'worker_threads'
import PQueue from 'p-queue'
import process from 'process'
import mongoose from 'mongoose'
import 'dotenv/config'

const queue = new PQueue({ concurrency: 5 })

const checkFinishedJob = () =>
  // logic goes here
  true

;(() => {
  if (parentPort) {
    await mongoose.connect(process.env.MONGODB_URL as string)

    setInterval(() => {
      queue.add(() => checkFinishedJob())
    }, 1000) // run each second
  } else {
    process.exit(0)
  }
})()