Queue doesnt process all added jobs in a consistent manner.

[udaya-reddy]

Description

Queue processing is not 100% consistent, leading to reliability issue of job execution. Submitting this as a bug.

Minimal, Working Test code to reproduce the issue.

Please see below.

(An easy to reproduce test case will dramatically decrease the resolution time.)

when concurrency is not specified, this occurs more frequently.

Bull version

4.11.5

Additional information

Hi,
This issue seem to still occur.
I am using Bull 4.11.5 and I use the queue for delayed jobs. sometimes the queue.process function gets called for most of queued jobs, but other times it doesnt. The job however is marked as completed and not removed from the completedJobs list. The ones where queue.process is called the jobs are not in queue as these are auto removed once completed.
I use default settings ( not providing any overrides ) on the queue.
`

          const redisOptions: any = {
  	host: config.redis.url,
  	port: config.redis.port,
  	password: config.redis.password,
  }
  if (config.redis.port != 6379 && config.redis.port != '6379') {
  	redisOptions.tls = true
  	redisOptions.enableTLSForSentinelMode = false
  }
           queue = new Queue(`scheduler`, {
  	redis: redisOptions,
  })

`

				queue.process(config.queueConcurrency || 20, async (job, done) => {
					logger.debug(`Process: Event ${job.data.name} triggered: ${job.data.id}. Proceeding to confirm.`)
      done & done()
})

queue.add(event, { delay, jobId: event.id })

for adding delayed events. they also show in the queue as delayed where queried.
Issue is a few jobs move from delayed to Completed without the queue.process callback function being called at all.
Point to note: Redis connection may be getting reconnected only keepAlive time (defaults) is over. not sure if this could be a cause.

1. Is this a bug, where the status is in "completed" state, but queue.process function wasnt called? at the least they should be in "waiting" state if not yet processed.
2. or, is there something else that is being missed here.

[udaya-reddy]

there was another worker attached from another service. Closing