RedisManager does not appear to be updating in test env

Question

RedisManager does not appear to be updating in test env

shawnpyle opened this issue a year ago · comments

I would like to make convenience method enabled? on my jobs so that my app can determine if recurring jobs are processing like normal or show a notification that it needs attention.

An example class:

class Noop
  include ::Sidekiq::Job
  def perform; end
  def enabled?
     Sidekiq::Scheduler.instance.job_enabled?("noop")
  end
end

I am seeing different results for job_enabled?("noop") depending on the environment.

In development I see:

Sidekiq.schedule = { "noop" => { "every" => "1 minute", "class" => Noop.name, "enabled" => true } }
Sidekiq::Scheduler.instance.load_schedule!
puts Sidekiq.get_schedule
Noop.enabled? #=> **true**

In my test environment (bundle exec RAILS_ENV=test rails c), I'm seeing Noop.enabled? return false.

Sidekiq.schedule = { "noop" => { "every" => "1 minute", "class" => Noop.name, "enabled" => true } }
Sidekiq::Scheduler.instance.load_schedule!
puts Sidekiq.get_schedule
Noop.enabled? #=> **false**

It appears the RedisManager is not being updated when I'm asking it to load_schedule! and since job_enabled? looks there first, it's returning false.

Is this expected?
Is there a better way to be loading these schedules for testing?

Bundle Env

$> bundle env
Bundler       2.4.13
  Platforms   ruby, x86_64-darwin-21
Ruby          3.0.6p216 (2023-03-30 revision 23a532679b406cb53c0edfc00c91c32a5ccd335a) [x86_64-darwin-21]
  Full Path   /Users/spyle/.asdf/installs/ruby/3.0.6/bin/ruby
  Config Dir  /Users/spyle/.asdf/installs/ruby/3.0.6/etc
RubyGems      3.4.13
  Gem Home    /Users/spyle/.asdf/installs/ruby/3.0.6/lib/ruby/gems/3.0.0
  Gem Path    /Users/spyle/.gem/ruby/3.0.0:/Users/spyle/.asdf/installs/ruby/3.0.6/lib/ruby/gems/3.0.0
  User Home   /Users/spyle
  User Path   /Users/spyle/.gem/ruby/3.0.0
  Bin Dir     /Users/spyle/.asdf/installs/ruby/3.0.6/bin
Tools
  Git         2.37.1 (Apple Git-137.1)
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Built At          2023-05-10
Git SHA           26eb456c6c
Released Version  true

Sidekiq Config

config/sidekiq.yml

# frozen_string_literal: true

# Sample configuration file for Sidekiq.
# Options here can still be overridden by cmd line args.
# Place this file at config/sidekiq.yml and Sidekiq will
# pick it up automatically.
---
:verbose: true
# concurrency should match database.yml pool
:concurrency: 5
# :timeout: 25
# :logfile: ./log/sidekiq.log
# :pidfile: ./tmp/pids/sidekiq.pid

# The listing of queues is order specific, with the highest priority queues at the top.
# https://tosbourn.com/dedicated-sidekiq-queue/
:queues:
  - credential
  - syncing
  - default
  # - mailer # TODO

# you can override concurrency based on environment
# production:
#   :concurrency: 25
# staging:
#   :concurrency: 15

# https://github.com/sidekiq-scheduler/sidekiq-scheduler#configuration-options
:scheduler:
  :enabled: true

config/initializers/sidekiq.rb

# frozen_string_literal: true

require "sidekiq"
require "sidekiq/web"
require "sidekiq-scheduler"
require "sidekiq-scheduler/web"
require "sidekiq-failures"

Sidekiq.configure_server do |config|
  config.redis = {
    url: ENV["REDIS_URL"]
  }

  # Load the schedules from the scheduler file.
  # The :schduler: > :schedule: configuration in config/sidekiq.yml would not load the schedule. Loading it from a separate file was the only working solution.
  # https://github.com/sidekiq-scheduler/sidekiq-scheduler#load-the-schedule-from-a-different-file
  config.on(:startup) do
    Sidekiq.schedule = YAML.load_file(File.expand_path("../../sidekiq_scheduler.yml", __FILE__))
    Sidekiq::Scheduler.instance.load_schedule!
  end

  # Add SentryErrorLogger to lib/sidekiq/middleware/server/
  # Register SentryErrorLogger in `config/initializers/sidekiq.rb`
  # See https://medium.com/clarisights/monitoring-and-reporting-errors-from-sidekiq-jobs-4b946036c9f4
  # config.server_middleware do |chain|
  #   chain.add Sidekiq::Middleware::Server::SentryErrorLogger
  # end
end

# Disable the max count of failures so we capture them all
# https://github.com/mhfs/sidekiq-failures#configuring
# https://github.com/mhfs/sidekiq-failures/issues/146#issuecomment-1364139520
Sidekiq.failures_max_count = false

Sidekiq::Web.use Rack::Auth::Basic do |username, password|
  # Protect against timing attacks:
  # - See https://codahale.com/a-lesson-in-timing-attacks/
  # - See https://thisdata.com/blog/timing-attacks-against-string-comparison/
  # - Use & (do not use &&) so that it doesn't short circuit.
  # - Use digests to stop length information leaking (see also ActiveSupport::SecurityUtils.variable_size_secure_compare)
  ActiveSupport::SecurityUtils.secure_compare(::Digest::SHA256.hexdigest(username), ::Digest::SHA256.hexdigest(Rails.application.credentials.config[:sidekiq][:web][:username])) &
    ActiveSupport::SecurityUtils.secure_compare(::Digest::SHA256.hexdigest(password), ::Digest::SHA256.hexdigest(Rails.application.credentials.config[:sidekiq][:web][:password]))
end if Rails.env.production? || Rails.env.qa?

Marcelo Lauxen · Answer 1 · Sat Jul 01 2023 04:05:34 GMT+0800 (China Standard Time)

when you run Sidekiq::Scheduler.instance.load_schedule! in test mode, what does the log output? Does it output any of those of this method?

Shawn Pyle · Answer 2 · Sun Jul 02 2023 08:00:08 GMT+0800 (China Standard Time)

I get a disabled message:

$> bin/spring stop; RAILS_ENV=test bundle exec rails c
Spring stopped.
Running via Spring preloader in process 53352
Loading test environment (Rails 6.1.7.3)
irb(main):001:1* class Noop
irb(main):002:1*   include ::Sidekiq::Job
irb(main):003:1*   def perform; end
irb(main):004:2*   class << self
irb(main):005:3*     def enabled?
irb(main):006:3*        Sidekiq::Scheduler.instance.job_enabled?("noop")
irb(main):007:2*     end
irb(main):008:1*   end
irb(main):009:0> end
=> :enabled?
irb(main):010:0> Sidekiq.schedule = { "noop" => { "every" => "1 minute", "class" => Noop.name, "enabled" => true } }
2023-07-01T23:59:11.038Z pid=53352 tid=12vw INFO: Sidekiq 7.1.1 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>"redis://:REDACTED@localhost:8212/1"}
=> {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true}}
irb(main):011:0>
irb(main):012:0> Sidekiq::Scheduler.instance.reload_schedule!
2023-07-01T23:59:16.219Z pid=53352 tid=12vw INFO: SidekiqScheduler is disabled
=> true
irb(main):013:0> puts Sidekiq.get_schedule
{"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
=> nil
irb(main):014:0> Noop.enabled? #=> true in development, false in test
=> false

I also get the message SidekiqScheduler is disabled in development too. However, job_enabled? returns true.

$> bin/spring stop; RAILS_ENV=development bundle exec rails c
Spring stopped.
Running via Spring preloader in process 52706
Loading development environment (Rails 6.1.7.3)
[1] pry(main)> class Noop
  include ::Sidekiq::Job
  def perform; end
  class << self
    def enabled?
       Sidekiq::Scheduler.instance.job_enabled?("noop")
    end
  end
[1] pry(main)> class Noop
  include ::Sidekiq::Job
  def perform; end
  class << self
    def enabled?
      Sidekiq::Scheduler.instance.job_enabled?("noop")
    end
  end
end

=> :enabled?
[3] pry(main)> Sidekiq.schedule = { "noop" => { "every" => "1 minute", "class" => Noop.name, "enabled" => true } }

=> {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true}}
[4] pry(main)> Sidekiq::Scheduler.instance.reload_schedule!

2023-07-01T23:57:29.159Z pid=52706 tid=16xi INFO: SidekiqScheduler is disabled
=> true
[5] pry(main)> puts Sidekiq.get_schedule

{"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
=> nil
[6] pry(main)> Noop.enabled? #=> true in development, false in test

=> true

Marcelo Lauxen · Answer 3 · Mon Jul 03 2023 21:52:02 GMT+0800 (China Standard Time)

@shawnpyle You did use reload_schedule! in your last example, in the first one, you used load_schedule!, could you use reload_schedule! in the test and see what you get? load_schedule! checks if the scheduler is enabled and then loads it, while the reload_schedule! doesn't, it only loads the schedule from Redis. I wonder if it is that

Shawn Pyle · Answer 4 · Fri Jul 07 2023 03:33:33 GMT+0800 (China Standard Time)

@marcelolx Yes, reload_schedule! did work for this case. I could have sworn I did try that but it is reproducible now. Thank you!

$> bin/spring stop; RAILS_ENV=test bundle exec rails c
Spring stopped.
Running via Spring preloader in process 39188
Loading test environment (Rails 6.1.7.3)
irb(main):001:1* class Noop
irb(main):002:1*   include ::Sidekiq::Job
irb(main):003:1*   def perform; end
irb(main):004:2*   class << self
irb(main):005:3*     def enabled?
irb(main):006:3*        Sidekiq::Scheduler.instance.job_enabled?("noop")
irb(main):007:2*     end
irb(main):008:1*   end
irb(main):009:0> end
=> :enabled?
irb(main):010:0>
irb(main):011:0> Sidekiq.schedule = { "noop" => { "every" => "1 minute", "class" => Noop.name, "enabled" => true } }
2023-07-06T19:27:33.399Z pid=39188 tid=rz4 INFO: Sidekiq 7.1.1 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>"redis://:REDACTED@localhost:8212/1"}
=> {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true}}
irb(main):012:0> Sidekiq::Scheduler.instance.reload_schedule!
2023-07-06T19:27:35.861Z pid=39188 tid=rz4 INFO: SidekiqScheduler is disabled
=> true
irb(main):013:0> puts Sidekiq.get_schedule
{"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
=> nil
irb(main):014:0> Noop.enabled?
=> true

Shawn Pyle · Answer 5 · Thu Jul 20 2023 01:30:52 GMT+0800 (China Standard Time)

@marcelolx Did some additional testing with by scripting to try to make a more reproducible scenario.

Here is the script I am using:

# sidekiq_scheduler_enable.rb

class Noop
  include ::Sidekiq::Job
  def perform; end
end

SCHEDULE_NAME = "noop"
SCHEDULE = { SCHEDULE_NAME => { "every" => "1 minute", "class" => Noop.name, "enabled" => true } }

def enabled?
  Sidekiq::Scheduler.instance.job_enabled?(SCHEDULE_NAME).inspect
rescue => e
  e.message
end

def checks(header:)
	puts "\n### #{header} ###"
	puts "Scheduler Instance ID: #{Sidekiq::Scheduler.instance.object_id}"
	puts "Schedule: #{Sidekiq.schedule.inspect}"
	puts "Schedule state: #{Sidekiq::Scheduler.instance.send(:schedule_state, SCHEDULE_NAME)}"
	puts "Noop enabled? #{enabled?}" #=> true in development, false in test
end

checks(header: "BEFORE")

Sidekiq.schedule = SCHEDULE
checks(header: "SCHEDULED")

Sidekiq::Scheduler.instance.reload_schedule!
checks(header: "RELOAD 1")
# Expect Noop to be enabled but doesn't work in test.

Sidekiq.schedule = {}
checks(header: "UNLOADED")

Sidekiq::Scheduler.instance.reload_schedule!
checks(header: "RELOAD 2")

Sidekiq.schedule = SCHEDULE
checks(header: "RESCHEDULED")

Sidekiq::Scheduler.instance.reload_schedule!
checks(header: "RELOAD 3")

In development, the Noop enabled? is true as expected in the RELOAD 1 and RELOAD 3 sections.

$> bin/spring stop; RAILS_ENV=development bundle exec rails runner script/sidekiq_scheduler_enable.rb
Spring stopped.
Running via Spring preloader in process 93563

### BEFORE ###
Scheduler Instance ID: 26880
Schedule: nil
2023-07-19T15:47:26.973Z pid=93563 tid=22mz INFO: Sidekiq 7.1.1 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>"redis://:REDACTED@localhost:8212/0"}
Schedule state: {}
Noop enabled? undefined method `[]' for nil:NilClass

### SCHEDULED ###
Scheduler Instance ID: 26880
Schedule: {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
Schedule state: {}
Noop enabled? true
2023-07-19T15:47:26.997Z pid=93563 tid=22mz INFO: SidekiqScheduler is disabled

### RELOAD 1 ###
Scheduler Instance ID: 26880
Schedule: {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
Schedule state: {}
Noop enabled? true

### UNLOADED ###
Scheduler Instance ID: 26880
Schedule: {}
Schedule state: {}
Noop enabled? nil
2023-07-19T15:47:27.014Z pid=93563 tid=22mz INFO: SidekiqScheduler is disabled

### RELOAD 2 ###
Scheduler Instance ID: 26880
Schedule: {}
Schedule state: {}
Noop enabled? nil

### RESCHEDULED ###
Scheduler Instance ID: 26880
Schedule: {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
Schedule state: {}
Noop enabled? true
2023-07-19T15:47:27.119Z pid=93563 tid=22mz INFO: SidekiqScheduler is disabled

### RELOAD 3 ###
Scheduler Instance ID: 26880
Schedule: {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
Schedule state: {}
Noop enabled? true

However in the test environment, it never returns true, even after reloading the schedule.

$> bin/spring stop; RAILS_ENV=test bundle exec rails runner script/sidekiq_scheduler_enable.rb
Spring stopped.
Running via Spring preloader in process 32848

### BEFORE ###
Scheduler Instance ID: 18760
Schedule: nil
2023-07-19T17:26:40.382Z pid=32848 tid=tf4 INFO: Sidekiq 7.1.1 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>"redis://:REDACTED@localhost:8212/1"}
Schedule state: {"enabled"=>false}
Noop enabled? undefined method `[]' for nil:NilClass

### SCHEDULED ###
Scheduler Instance ID: 18760
Schedule: {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
Schedule state: {"enabled"=>false}
Noop enabled? false
2023-07-19T17:26:40.396Z pid=32848 tid=tf4 INFO: SidekiqScheduler is disabled

### RELOAD 1 ###
Scheduler Instance ID: 18760
Schedule: {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
Schedule state: {"enabled"=>false}
Noop enabled? false

### UNLOADED ###
Scheduler Instance ID: 18760
Schedule: {}
Schedule state: {"enabled"=>false}
Noop enabled? nil
2023-07-19T17:26:40.403Z pid=32848 tid=tf4 INFO: SidekiqScheduler is disabled

### RELOAD 2 ###
Scheduler Instance ID: 18760
Schedule: {}
Schedule state: {"enabled"=>false}
Noop enabled? nil

### RESCHEDULED ###
Scheduler Instance ID: 18760
Schedule: {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
Schedule state: {"enabled"=>false}
Noop enabled? false
2023-07-19T17:26:40.412Z pid=32848 tid=tf4 INFO: SidekiqScheduler is disabled

### RELOAD 3 ###
Scheduler Instance ID: 18760
Schedule: {"noop"=>{"every"=>"1 minute", "class"=>"Noop", "enabled"=>true, "queue"=>"default"}}
Schedule state: {"enabled"=>false}
Noop enabled? false

The above happens most often but is not deterministic. In some cases (I'm working to identify the scenario) job_enabled? will return true when expected. Do you know of any reason why Sidekiq::Scheduler.instance.schedule_state would return {"enabled"=>false} in some cases?

Shawn Pyle · Answer 6 · Thu Jul 20 2023 04:42:21 GMT+0800 (China Standard Time)

Looking at how the scheduler is tested, it appears that a Sidekiq.redis(&:flushall) goes a long way to clear up some of the issues I was having. I still think there is some issue with what RedisManager holds on to regarding enabled state but for now, I'm out of the woods.