How does delayed compare to good_job?
rgaufman opened this issue · comments
I am looking for a Sidekiq alternative and from what I can tell this is quite similar to good_job? - Could you please comment where you see the differences/advantages?
The lack of a dashboard in delayed is currently the reason that is making me hesitant, is that something on the roadmap or is there an immediate solution for that?
I've used good_job previously but as I'm experimenting moving Postgres -> SQLite for most cases I need a job backend that supports SQLite... doesn't seem there are many out there but this is one of them
This is a very good point, we recently switched from MariaDB to Postgres specifically because many gems (including good_job) require it. It is a big advantage that this gem supports other databases.
We've been planning to put a feature matrix into the README to help clarify some of this but the in general our philosophy drives us to the most constrained solution that can reasonably work for general purpose background workloads (though obviously through Betterment's lens). We use it and rely upon it for our most important stuff at pretty significant scale.
On the topic of dashboarding, we've got some stuff that we've been considering tidying up for public consumption, so it's great to hear it's important to you for adoption.
see also this thread on dashboarding - feel free to add or upvote specifics you're looking for in dashboarding.
My initial drive to good_job was the strong DB transaction guarantees for queueing jobs that you don't have out of the box with separate queuing mechanisms like Redis (unless you have some transactionally staged job queues)
As I don't need to process thousands of jobs per second I prefer having my job queue in the same DB with transactional guarantees AND not having to run a separate queueing backend, i.e. Redis
For us, one of the advantages of delayed is that it does not need LISTEN/NOTIFY or advisory locks because we need to run pg_bouncer in transaction mode.
Due to how we deploy our application, there are many server and worker processes, each having an ActiveRecord pool. They can easily open up to 2000-3000 database connection but are mostly idle. With pg_bouncers transaction mode, the actual active server connection count can be reduced to ~20, but some connection-level features are not available.
For that reason, we cannot use good_job, but only delayed, that does efficient polling only.
We are still testing a few things, especially with the job timeout things because we have some jobs that can run for hours or days, and are looking into unique job keys to restrict concurrent scheduling. We would like to use this to schedule cron jobs with a distributed scheduler.
That is why we are testing with delayed now. We definitely prefer the co-transactionality over our current sidekiq cluster too! Hopefully, delayed will do a good job for us soon too ;)
For us, one of the advantages of delayed is that it does not need LISTEN/NOTIFY or advisory locks because we need to run pg_bouncer in transaction mode.
Due to how we deploy our application, there are many server and worker processes, each having an ActiveRecord pool. They can easily open up to 2000-3000 database connection but are mostly idle. With pg_bouncers transaction mode, the actual active server connection count can be reduced to ~20, but some connection-level features are not available.
For that reason, we cannot use good_job, but only delayed, that does efficient polling only.
We are still testing a few things, especially with the job timeout things because we have some jobs that can run for hours or days, and are looking into unique job keys to restrict concurrent scheduling. We would like to use this to schedule cron jobs with a distributed scheduler.
That is why we are testing with delayed now. We definitely prefer the co-transactionality over our current sidekiq cluster too! Hopefully, delayed will do a good job for us soon too ;)
How did things fare with usage of delayed so far? We're exploring new solutions and the pg_bouncer constraint is something we have too!