Sidekiq 7.0 support?

Question

Sidekiq 7.0 support?

aglushkov opened this issue 2 years ago · comments

I have this error when after upgrading sidekiq to 7.0

no implicit conversion of Sidekiq::Config into Hash

Backtrace

2022-10-14T10:04:34.670Z pid=2228626 tid=1byma INFO: Sidekiq 7.0.0.beta1 connecting to Redis with options {:host=>"localhost", :db=>"0", :size=>1, :pool_name=>"default", :url=>nil}
2022-10-14T10:04:34.673Z pid=2228626 tid=1byma WARN: {"context":"Exception during Sidekiq lifecycle event.","event":"startup"}
2022-10-14T10:04:34.673Z pid=2228626 tid=1byma WARN: TypeError: no implicit conversion of Sidekiq::Config into Hash
2022-10-14T10:04:34.674Z pid=2228626 tid=1byma WARN: lib/ruby/gems/3.1.0/gems/sidekiq-scheduler-4.0.3/lib/sidekiq-scheduler/manager.rb:40:in `merge'
lib/ruby/gems/3.1.0/gems/sidekiq-scheduler-4.0.3/lib/sidekiq-scheduler/manager.rb:40:in `load_scheduler_options'
lib/ruby/gems/3.1.0/gems/sidekiq-scheduler-4.0.3/lib/sidekiq-scheduler/manager.rb:21:in `initialize'
lib/ruby/gems/3.1.0/gems/sidekiq-scheduler-4.0.3/lib/sidekiq-scheduler.rb:21:in `new'
lib/ruby/gems/3.1.0/gems/sidekiq-scheduler-4.0.3/lib/sidekiq-scheduler.rb:21:in `block (2 levels) in <main>'
lib/ruby/gems/3.1.0/gems/sidekiq-7.0.0.beta1/lib/sidekiq/component.rb:58:in `block in fire_event'
lib/ruby/gems/3.1.0/gems/sidekiq-7.0.0.beta1/lib/sidekiq/component.rb:57:in `each'
lib/ruby/gems/3.1.0/gems/sidekiq-7.0.0.beta1/lib/sidekiq/component.rb:57:in `fire_event'
lib/ruby/gems/3.1.0/gems/sidekiq-7.0.0.beta1/lib/sidekiq/cli.rb:105:in `run'
lib/ruby/gems/3.1.0/gems/sidekiq-7.0.0.beta1/bin/sidekiq:31:in `<top (required)>'
bin/sidekiq:25:in `load'

Marcelo Lauxen · Answer 1 · Fri Oct 14 2022 22:43:05 GMT+0800 (China Standard Time)

Yeah, no support yet... We probably will have to make some core changes to support sidekiq 7

Oskar Eichler · Answer 2 · Fri Oct 28 2022 01:26:44 GMT+0800 (China Standard Time)

That would be highly appreciated now that Sidekiq 7 has officially been released! 🙏🏼

Tony Drake · Answer 3 · Sat Oct 29 2022 23:44:48 GMT+0800 (China Standard Time)

I have a very hacky-but-appears-to-function branch that (with a minimal Rails 7 app + sidekiq 7) appears to run the scheduler as expected (at least for my use cases).

https://github.com/t27duck/sidekiq-scheduler/tree/sidekiq7

Specs are blowing up and not 100% happy with the hack on top of the previous hack I did to work around Sidekiq's internals changes, but this is at least a starting point.

@marcelolx if you want, I can make my branch a WIP/POC PR to get other's eyes and opinions on it.

Marcelo Lauxen · Answer 4 · Mon Oct 31 2022 19:05:48 GMT+0800 (China Standard Time)

@t27duck a PR is more than welcome, we can iterate on top of it. I'm short on time so any help/progress is appreciated!

Shayon Mukherjee · Answer 5 · Mon Oct 31 2022 23:06:29 GMT+0800 (China Standard Time)

Happy to attempt at a PR too if any further bandwidth is needed @t27duck @marcelolx

Tony Drake · Answer 6 · Wed Nov 02 2022 05:40:53 GMT+0800 (China Standard Time)

I went and made a PR based on my branch. Obviously, things are still broken in CI.

Day job is a bit hectic for the start of my week, but later on I should be able to take a stab at making it better.

Probably worth moving discussion into that for PR for now?

Marcelo Lauxen · Answer 7 · Wed Nov 02 2022 18:21:11 GMT+0800 (China Standard Time)

Definitely @t27duck, and thanks for coming up with a PR!

Ryan Wilson · Answer 8 · Fri Nov 04 2022 00:29:54 GMT+0800 (China Standard Time)

Thank you so much for your work on this!

Marcelo Lauxen · Answer 9 · Fri Nov 04 2022 18:31:56 GMT+0800 (China Standard Time)

hey everyone, @t27duck has done an amazing job with the PR and we have already merged it. Although, we won't cut a release yet because we're working on a refactor to make some things simpler and less painful to deal with (for supporting different Sidekiq versions), if you'll want to keep an eye on it or have suggestions, feel free to comment on #412

While no release is made, you can point sidekiq-scheduler in your Gemfile to the following ⬇️ I recommend pointing to that exact commit because if #412 gets merged, there may be some breaking changes that will make your scheduler stop working.

gem "sidekiq-scheduler", github: "sidekiq-scheduler/sidekiq-scheduler", branch: "master", ref: "76573dbc46c73afa213331049cfc5657582dc376"

Marcelo Lauxen · Answer 10 · Fri Dec 23 2022 19:15:51 GMT+0800 (China Standard Time)

Hello everyone, we have released a beta version of sidekiq-scheduler with support for Sidekiq 7 https://github.com/sidekiq-scheduler/sidekiq-scheduler/releases/tag/v5.0.0.beta1

Tony Drake · Answer 11 · Fri Dec 23 2022 20:33:24 GMT+0800 (China Standard Time)

May be a good idea to keep this open until v5 final of the gem goes out

Tony Drake · Answer 12 · Fri Dec 23 2022 20:34:10 GMT+0800 (China Standard Time)

Just deployed beta1 + sidekiq 7.0.2 on a side production app I have. No issues so far as far as I can see. Things are being scheduled as expected at the right time.

Marcelo Lauxen · Answer 13 · Sun Dec 25 2022 00:16:52 GMT+0800 (China Standard Time)

May be a good idea to keep this open until v5 final of the gem goes out

Sounds good to me @t27duck!

Magnus Bergmark · Answer 14 · Fri Dec 30 2022 15:49:26 GMT+0800 (China Standard Time)

I'm unable to boot my app using the beta1 release. I get this error:

/usr/local/bundle/gems/sidekiq-scheduler-5.0.0.beta1/lib/sidekiq-scheduler/manager.rb:1:in `require': cannot load such file -- redis (LoadError)
        from /usr/local/bundle/gems/sidekiq-scheduler-5.0.0.beta1/lib/sidekiq-scheduler/manager.rb:1:in `<top (required)>'
        from /usr/local/bundle/gems/sidekiq-scheduler-5.0.0.beta1/lib/sidekiq-scheduler.rb:6:in `require_relative'
        from /usr/local/bundle/gems/sidekiq-scheduler-5.0.0.beta1/lib/sidekiq-scheduler.rb:6:in `<top (required)>'

True enough, it uses require "redis" at the top of that file. Is there an undocumented dependency somewhere?

Gemfile.lock

    sidekiq (7.0.2)
      concurrent-ruby (< 2)
      connection_pool (>= 2.3.0)
      rack (>= 2.2.4)
      redis-client (>= 0.11.0)
    sidekiq-scheduler (5.0.0.beta1)
      rufus-scheduler (~> 3.2)
      sidekiq (>= 4, < 8)
      tilt (>= 1.4.0)

[EDIT:]

I added an empty lib/redis.rb file and the following patch to my config/boot.rb file in Rails:

 ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__)
 
+$LOAD_PATH << File.expand_path("../lib", __dir__)
+require "redis"
+
 require "bundler/setup" # Set up gems listed in the Gemfile.

Now the app boots and schedule seem to be working correctly.
Maybe this confirms that the dependency isn't used… 🤷

Tony Drake · Answer 15 · Sat Dec 31 2022 02:07:27 GMT+0800 (China Standard Time)

@Mange Does #416 work for you?

It does look like this was a hold over from an older version of the gem + when sidekiq explicitly only worked with the redis gem.

Marcelo Lauxen · Answer 16 · Mon Jan 02 2023 21:17:33 GMT+0800 (China Standard Time)

@Mange can you test with v5.0.0.beta2 release? It contains the changes of #416 that @t27duck mentioned

Magnus Bergmark · Answer 17 · Tue Jan 03 2023 23:16:40 GMT+0800 (China Standard Time)

Yes, I can confirm that beta2 worked out of the box without hacks! Thank you!

Marcelo Lauxen · Answer 18 · Tue Jan 03 2023 23:29:14 GMT+0800 (China Standard Time)

perfect, thanks for the feedback @Mange!

@t27duck Let's wait another week, then I think we're good to go with a final v5 release.

Laney Stroup · Answer 19 · Wed Jan 18 2023 13:31:55 GMT+0800 (China Standard Time)

I found this issue looking up the original error message above (no implicit conversion of Sidekiq::Config into Hash...). I am now running 5.0.0.beta2, and it is working perfectly.

Marlos · Answer 20 · Wed Jan 18 2023 19:42:27 GMT+0800 (China Standard Time)

5.0.0.beta2 working fine here too. 👍🏻

Marcelo Lauxen · Answer 21 · Wed Jan 18 2023 21:35:02 GMT+0800 (China Standard Time)

Thanks for the updates, I'll push the final release by the end of the week!

Tim Scott · Answer 22 · Thu Jan 19 2023 07:05:58 GMT+0800 (China Standard Time)

I tried 5.0.0.beta2 and I'm getting:

Schedule empty! Set Sidekiq.schedule

I put a breakpoint here and sidekiq_config appears to be the default schedule as if it's never reading config/sidekiq.yml.

I'm using Sidekiq.configure_embed to run sidekiq.

Tim Scott · Answer 23 · Thu Jan 19 2023 08:47:23 GMT+0800 (China Standard Time)

I'm using Sidekiq.configure_embed to run sidekiq.

I added the following to my config block, and it's working:

sidekiq_config = YAML.load_file('config/sidekiq.yml')
config.merge! sidekiq_config

I guess sidekiq 7 doesn't load config/sidekiq.yml when running embedded.

Marcelo Lauxen · Answer 24 · Thu Jan 19 2023 21:29:15 GMT+0800 (China Standard Time)

Doesn't this look more a problem with Sidekiq? If you set a configuration for sidekiq in the sidekiq.yml file, isn't that config being applied to sidekiq when it runs in embed mode? @timscott If you could come up with a sample app, or at least a sample of your sidekiq config file, so I can simulate it here, I would appreciate it

Marcelo Lauxen · Answer 25 · Mon Jan 23 2023 18:28:38 GMT+0800 (China Standard Time)

Hello everyone, sidekiq-scheduler v5 has been released https://github.com/sidekiq-scheduler/sidekiq-scheduler/releases/tag/v5.0.0, thanks to everyone involved, we really appreciate it!

@timscott I created a new issue for your problem #419, as soon as I have some time I'll investigate more and come up with a fix, thanks for reporting it.