Locks not being cleared on SIGKILL

[brett]

Our jobs are remaining locked after the worker process receives the SIGKILL (in our case from docker stop after the grace period elapses).

I'm not sure if this is by design or not, the README states (may being the keyword):

the process and may result in long-running jobs remaining locked until Delayed::Worker.max_run_time has elapsed.

In my tests they always remains locked when a job is running and Delayed also doesn't terminate with a SIGTERM.

Here is simple rake task I used to simulate the issue from Delayed

class TestWorker
  def start
    trap('TERM') { quit! }
    trap('INT') { quit! }

    500.times do |i|
      puts "Run #{i}"
      sleep 1.second
    end
  ensure
    on_exit!
  end

  def quit!
    puts 'quit!'
  end

  def on_exit!
    puts 'on_exit!'
  end
end

namespace :test do
  desc "Test signal interupts"
  task work: :environment do
    TestWorker.new.start
  end
end

If you run that as as a rake task, it won't terminate on a SIGTERM, only SIGKILL which won't execute the ensure block which in Delayed is what unlocks the jobs.

Whereas if I explicitly `exit` in the `quit!` method it works as expected.

class TestWorker
  def start
    trap('TERM') { quit! }
    trap('INT') { quit! }

    500.times do |i|
      puts "Run #{i}"
      sleep 1.second
    end
  ensure
    on_exit!
  end

  def quit!
    puts 'quit!'
    exit
  end

  def on_exit!
    puts 'on_exit!'
  end
end

namespace :test do
  desc "Test signal interupts"
  task work: :environment do
    TestWorker.new.start
  end
end

What's the recommended way to gracefully shut down a running job when the process is terminated? Should a job implement its own traps or is there a hook that Delayed offers?

• Rails 7.0.7
• Ruby 3.2.2
• Delayed 0.5.0

[Nathan Griffith]

Hi @suwyn,

Thanks for reaching out!

I've modified your example a bit to add a break if stop? check:

class TestWorker
  def start
    trap('TERM') { quit! }
    trap('INT') { quit! }

    loop do
      3.times do |i|
        puts "Run #{i}"
        sleep 1
      end
      break if stop?
    end
  ensure
    on_exit!
  end

  def stop?
    @stop
  end

  def quit!
    @stop = true
    puts 'quit!'
  end

  def on_exit!
    puts 'on_exit!'
  end
end

This more closely mirrors how delayed responds to a SIGINT or SIGTERM, by attempting to finish the current set of jobs assigned to the thread pool before exiting. (I'm simulating that with 3 "jobs," repeated forever by the "worker" in a loop.)

Here's a sample output with a SIGINT (Ctrl+C) sent during the second set of jobs:

$ rake test:work
Run 0
Run 1
Run 2
Run 0
^Cquit!
Run 1
Run 2
on_exit!

It loops around to pick up more jobs until it receives SIGINT/SIGTERM, at which point it will finish the current pool and exit cleanly instead of picking up more jobs. However, if it receives a SIGKILL instead, it will immediately exit, without attempting to clean anything up. (This signal is special and is handled by the kernel/OS -- the worker never actually receives it so there would be no way for it to react.)

What's the recommended way to gracefully shut down a running job when the process is terminated?

Right, so to answer your question, the way to gracefully shut down is to first send a SIGTERM, give jobs a chance to finish, and then send a SIGKILL later if/when you need to fully stop the process. If you're not seeing SIGTERM produce a clean exit within a reasonable amount of time, it's likely due to long-running jobs. So, this means that jobs need to be short-lived enough to complete gracefully within that waiting period, otherwise the worker will exit ungracefully and the new workers will wait until locked_at + Delayed::Worker.max_run_time in order to be sure that no other worker is running the job.

In general, I'd suggest deconstructing jobs into shorter units of work, but keep in mind that once you send the SIGTERM, you know that the worker won't pick up any new jobs, so—depending on your deployment infrastructure—you could wait a very long time before sending a SIGKILL! (Perhaps even the entire max_run_time, at which point you'll know for sure that all jobs have either completed or timed-out-with-cleanup.)

Should a job implement its own traps or is there a hook that Delayed offers?

I hadn't really considered this before. Generally I think we've found that keeping max_run_time configured to the default of 20 min (or less)—and waiting after SIGTERM for jobs to finish gracefully—has produced the best overall results (in addition to making sure that jobs are all idempotent & re-runnable). Even if a worker can't exit gracefully due to a long-running job, the longest we'd have to wait for that job to be picked up again is 20 minutes (and since this typically only affects jobs that take a long time to complete, we don't expect fast turnaround anyways).

[brett]

Thanks for the explanation @smudge it all make sense.

You're correct in that the culprit for us is a long running job and that it should be broken down into shorter units of work and to keep max_run_time sensible. We had been delaying that work and kept bumping the max_run_time up to compensate 😨 and given that we're cleaning up containers 60 seconds after a deploy those jobs are staying locked/stalled for up to 3hours before they get cleared and picked up again.

While we do have an option to trap the SIGTERM in the job itself, it would feel cleaner if that came from the Job API (e.g. perhaps rescue_from) but I find myself agreeing with you again here and want to avoid that complexity.

tl;dr - We'll decompose our long running job so that it runs in multiple units, allowing us to keep max_run_time low enough to allow containers to be cleaned up in a timely manner. Thanks!