Giters

xpepermint / mongodb-cron

MongoDB collection as crontab

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Question: Idempotent jobs?

batjko opened this issue · comments

commented

Couple of questions, all related to restarting instances:

The documentation says:

If cron is unexpectedly interrupted during the processing of a job (e.g. server shutdown), the system automatically recovers and transparently restarts.

  • Does this mean, when my app has a bunch of jobs defined and uses collection.create() every time it starts, it will not re-create those jobs, if they already exist?
  • How does mongodb-cron identify that the job I'm trying to create already exists? There doesn't seem to be an identifier I can set.
  • What about multiple instances, say if I have several replicas of my app running (e.g. in k8s or docker swarm)? Can mongodb-cron handle this seamlessly, when all the replicas are trying to access the job queue collection and process jobs at the same time?

Many thanks.

commented
  • True: You create a job document and the document will stay there until YOU say the job has completed. You can also set autoRemove which will remove completed jobs.
  • YOU create an identifier when you are inserting the job (custom fields are allowed and you can e.g. set unique index).
  • It's designed to work with clusters so multiple replicas are supported. Every job is locked which means only one process can handle it. Failed jobs are reprocessed based on lockDuration. Make sure you check all API parameters.
commented

@xpepermint Thanks for the quick response.
Ok that makes sense. It wasn't clear to me from the start how jobs are meant to look, how to use them to execute something etc.

I am now keeping a { [jobName]: <function> } map, so that I can add a name field to each job in the collection and execute the right ones when the scheduler processes the queue, which I take it is roughly how it's meant to be used.

I will also update my job creation (collection.insert) to rather be an upsert, based on the job name, which should deal with the idempotency need in a cluster setup.

Thanks for the advice.