Running a 2nd sidekiq process will overwrite schedule configuration stored in Redis

[Will Spurgin]

I've noticed that, if one runs a 2nd sidekiq process with a separate configuration, the recurring jobs tab gets emptied out in the web extension.

Please note that the schedule itself with rufus still continues to operate and queue the scheduled jobs on the correct intervals.

I believe this might be the cause of #293

Setup

Run a rails app with Sidekiq and sidekiq-scheduler. Mount the Sidekiq::Web UI with the sidekiq-scheduler/web required.

Include a config/sidekiq.yml with a configured schedule, e.g.,

:concurrency: 5
:queues:
  - critical
  - default
  - low
:schedule:
  do_something_every_minute:
    class: DoSomethingJob
    args: matey
    queue: :scheduler
    cron: '0 * * * * * America/Los_Angeles'
# ...

Next create another sidekiq configuration, this time without a schedule:

:concurrency: 5
:queues:
  - scheduler
  - other

Steps to reproduce

1. Start the rails server
2. Start a sidekiq process with the default configuration loaded (the one with the schedule)
3. Go to the Recurring Jobs tab in the Sidekiq Web UI, you should see the schedule like so:

4. Start the other sidekiq process loading the configuration without a schedule
5. Refresh the Recurring Jobs tab, you'll see the schedule disappear:

This bug is order dependent, if the process with the schedule boots last, the Recurring Jobs tab will show the schedule.

I made a reproducible app here: https://github.com/wspurgin/vanilla/tree/sidekiq-scheduler

Hypothesis

I believe it has to do with how the schedule is stored. Namely, that the key used to store and retrieve the schedules is just schedules and not namespaced to the booted Sidekiq process (e.g., by pid or full process name). Therefore the last booted sidkeiq process' "schedule" configuration wins out.

[Marcelo Lauxen]

I would say that this is just not a supported feature by sidekiq-scheduler and not a bug. Why do you need to run one process with the schedule and the other one not?

[Will Spurgin]

Hi @marcelolx 👋 Thanks for the speedy reply!

There are scenarios where, like those hinted at in the Sidekiq Wiki, a service or set of services might have a handful of sidekiq processes for handling different queues. Only one configuration may be the definitive source of the "schedule" and its setup.

This issue arises when those Sidekiq process share the same Redis instances (which again, I feel is a some what normal practice when using Sidekiq). It feels redundant to have every configuration define the same schedule, it become very hard to do so if, for instance, the sidekiq processes in question do not share the same codebase but share the same redis infrastructure.

[Will Spurgin]

Also, just to note in the contrived example I have, I know that if the second configuration had :enabled: false then this specific example would not happen 😄 Which probably solves the "issue" (I hear you that since its kinda an un-envisioned case you don't really consider it a bug) for most people (like those in #293). e.g.,

:concurrency: 5
:queues:
  - scheduler
  - other

## Scheduler setup Configuration
:enabled: false # prevent sidekiq-scheduler from trying to load an empty `schedule`

I think, since that "solves" this specific example, it's probably fine. Maybe, if anything, we can consider that if schedule (the key) is not present in the configuration, it's "disabled" by default?

[Marcelo Lauxen]

@wspurgin Particularly I would prefer to keep it explicit, this way who is gonna look at the config file will know that the scheduler is disabled (with dynamic schedules it would/could start disabled and then we would have to enable it when the schedule changes).

Also what I think that could help would be using https://github.com/resque/redis-namespace, this way each "schedule" would be namespaces and they wouldn't override, but I think disabling the schedule is way easier.

[Will Spurgin]

👍 Heard on the explicit enabled/disable flag. Maybe just a documentation enhancement then (again, if anything)

Namespaces would also solve this specific issue... but that would mean all of Sidekiq is namespaced which might not be as desirable. Disabling is definitely the easiest thing to do.

[Marcelo Lauxen]

@wspurgin Documentation enhancements are welcome! If you can add the scenario described above I'll be happy to merge! (and anything else you consider valuable)

Yeah, disabling definitely is the easiest solution

[Will Spurgin]

Thanks @marcelolx 👍 Opened #362 for it 😄 Let me know what you think.

[Ibrahim Awwal]

Hmm, quick question: the key is :enabled: at the root level, right? Would it make more sense to have it under the :schedule: key or something?

Also, would you run into this issue if using a different file for the schedule? I'm thinking not since in that case each process would load it from the same file, but wanted to double-check. https://github.com/sidekiq-scheduler/sidekiq-scheduler#load-the-schedule-from-a-different-file

Thanks!

[Marcelo Lauxen]

@ibrahima Yes, it is at the root level. In this PR #412 we're going to move it to the scheduler key, which still is kind of root level. Moving under the schedule config doesn't "solves" the problem, since you still have to define it, right? I think what was expected was sidekiq-scheduler being disabled if no schedules key is present, but I think that is a bit dangerous (breaking change btw) since some people may be doing things with the assumption the scheduler will be enabled if not explicitly disabled.

Yes, it will because the issue is the schedules key in Redis being the same. If the docs are not clear about that, we can definitely improve them, but I think we should keep it clear that if you have to do that, you want to disable the scheduler explicitly.

[Ibrahim Awwal]

Ah yeah that makes sense re: the "root" key.

So just to confirm: If I am loading the Sidekiq-scheduler config from a different file, all of my Sidekiq processes will have the same sidekiq-scheduler config, even if they have different Sidekiq configs. This is different from some processes having a config defined and some having :enabled: false set since in the latter case not all processes load the config. But will loading the config from a different file cause any problems? Since each process will end up storing the same config over and over again it should be fine right? To me this seems to be the same as having multiple Sidekiq processes all using the same sidekiq.yml which defines the same schedule, which should be pretty normal. It's just that if we have multiple sidekiq yml config files then we run into this issue? Thanks for the clarification!