Add cron support via --with-scheduler

[Michael Angeletti]

@selwin since I'm poking around on here, I'd like to pick your brain / propose a feature:

Running rqworker with the --with-scheduler option is tremendous (enqueue_at and enqueue_in are great).

However (and this is assuming that I haven't misunderstood the docs), setting up cron tasks still also requires rq-scheduler. What would it take to support running a single manage.py rqworker --with-scheduler process to get everything (enqueue_at, enqueue_in, and cron)? I'd be happy to help with this, if it's something you think would fit within the system.

This would be really great, especially when running in a Docker container (vs needing Supervisor to run both).

P.S. sorry for the duplicate post (wrong user)

[Selwin Ong]

However (and this is assuming that I haven't misunderstood the docs), setting up cron tasks still also requires rq-scheduler.

This is correct.

What would it take to support running a single manage.py rqworker --with-scheduler process to get everything (enqueue_at, enqueue_in, and cron)? I'd be happy to help with this, if it's something you think would fit within the system.

I'd also love to bring in cron capabilities into RQ's core. The problem with the current cron system in RQ scheduler is that at times the same job can be executed simultaneously by two different workers. For example, if I have a job that runs for 5 minutes each, but is cronned to run once every minute.

RQ jobs are not designed to be run by different workers simultaneously.

So to solve this, what we'll need to have in RQ is some kind of job template system that when cron'ed will create actual jobs to be picked up by workers.

[Nikhil Verma]

Does that mean if there is just a single worker scheduling cron jobs would be simple? If so, how?

[Selwin Ong]

There's actually some progress on this. I decided to tackle this with a different approach. This PR will allow jobs to track multiple simultaneous executions. Once this is done, we can start planning on bringing cron like capabilities into RQ.

[Ernesto Revilla]

After seeing that cron like jobs could only be registered using rq-scheduler, I found that I couldn't seem them in the django-rq admin interface (which states explicitly that rq-scheduler jobs are not displayed). After diving through source code I found that rq-scheduler and rq.scheduler use different redis keys to store the job of scheduled jobs.

Would it be enough to change the rq-scheduler Redis key so the jobs would be displayed using django-rq in the Scheduled Jobs section?