Heroku shutdown issue when using workers since upgrading to Rails 7.0.7

Question

Heroku shutdown issue when using workers since upgrading to Rails 7.0.7

schinery opened this issue 10 months ago · comments

Describe the bug

Since upgrading to Rails 7.0.7 I'm getting a ThreadError can't be called from trap context error thrown in Heroku apps, which I've tracked down (at least I think I have) to an ActiveSupport deprecation warning being thrown in the #close_binder_listeners method when it tries to format the timestamp in the shutdown message.

This is what I get when the app restarts, so Puma shutdowns, on Heroku:

2023-08-16T12:24:23.518278+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2023-08-16T12:24:23.566145+00:00 app[web.1]: bundler: failed to load command: puma (/app/vendor/bundle/ruby/3.2.0/bin/puma)
2023-08-16T12:24:23.566355+00:00 app[web.1]: [9] ! Detected parent died, dying
2023-08-16T12:24:23.566593+00:00 app[web.1]: [6] ! Detected parent died, dying
2023-08-16T12:24:23.616726+00:00 app[web.1]: I, [2023-08-16T12:24:23.616376 #2]  INFO -- [Bugsnag]: Notifying https://notify.bugsnag.com of ThreadError
2023-08-16T12:24:23.617775+00:00 app[web.1]: W, [2023-08-16T12:24:23.617716 #2]  WARN -- [Bugsnag]: Waiting for 1 outstanding request(s)
2023-08-16T12:24:23.855340+00:00 app[web.1]: D, [2023-08-16T12:24:23.855199 #2] DEBUG -- [Bugsnag]: Request to https://notify.bugsnag.com completed, status: 200
2023-08-16T12:24:23.855593+00:00 app[web.1]: /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/mutex_m.rb:79:in `synchronize': can't be called from trap context (ThreadError)
2023-08-16T12:24:23.855594+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/mutex_m.rb:79:in `mu_synchronize'
2023-08-16T12:24:23.855594+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/notifications/fanout.rb:110:in `listeners_for'
2023-08-16T12:24:23.855595+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/notifications/fanout.rb:118:in `listening?'
2023-08-16T12:24:23.855595+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/notifications.rb:205:in `instrument'
2023-08-16T12:24:23.855596+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/deprecation/behaviors.rb:39:in `block in <class:Deprecation>'
2023-08-16T12:24:23.855596+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/deprecation/reporting.rb:26:in `block (2 levels) in warn'
2023-08-16T12:24:23.855596+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/deprecation/reporting.rb:26:in `each'
2023-08-16T12:24:23.855597+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/deprecation/reporting.rb:26:in `block in warn'
2023-08-16T12:24:23.855597+00:00 app[web.1]: from <internal:kernel>:90:in `tap'
2023-08-16T12:24:23.855598+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/deprecation/reporting.rb:22:in `warn'
2023-08-16T12:24:23.855598+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/deprecation/instance_delegator.rb:21:in `warn'
2023-08-16T12:24:23.855598+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/deprecation/instance_delegator.rb:28:in `warn'
2023-08-16T12:24:23.855598+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/core_ext/time/deprecated_conversions.rb:15:in `to_s'
2023-08-16T12:24:23.855599+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/puma-6.3.0/lib/puma/launcher.rb:220:in `close_binder_listeners'
2023-08-16T12:24:23.855599+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/puma-6.3.0/lib/puma/cluster.rb:327:in `block in setup_signals'
2023-08-16T12:24:23.855600+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/puma-6.3.0/lib/puma/cluster.rb:438:in `wait_readable'
2023-08-16T12:24:23.855600+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/puma-6.3.0/lib/puma/cluster.rb:438:in `run'
2023-08-16T12:24:23.855600+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/puma-6.3.0/lib/puma/launcher.rb:194:in `run'
2023-08-16T12:24:23.855600+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/puma-6.3.0/lib/puma/cli.rb:75:in `run'
2023-08-16T12:24:23.855601+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/gems/puma-6.3.0/bin/puma:10:in `<top (required)>'
2023-08-16T12:24:23.855601+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/bin/puma:25:in `load'
2023-08-16T12:24:23.855601+00:00 app[web.1]: from /app/vendor/bundle/ruby/3.2.0/bin/puma:25:in `<top (required)>'
2023-08-16T12:24:23.855602+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/cli/exec.rb:58:in `load'
2023-08-16T12:24:23.855602+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/cli/exec.rb:58:in `kernel_load'
2023-08-16T12:24:23.855602+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/cli/exec.rb:23:in `run'
2023-08-16T12:24:23.855602+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/cli.rb:492:in `exec'
2023-08-16T12:24:23.855602+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
2023-08-16T12:24:23.855603+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
2023-08-16T12:24:23.855603+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
2023-08-16T12:24:23.855603+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/cli.rb:34:in `dispatch'
2023-08-16T12:24:23.855603+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
2023-08-16T12:24:23.855604+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/cli.rb:28:in `start'
2023-08-16T12:24:23.855604+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.10/libexec/bundle:45:in `block in <top (required)>'
2023-08-16T12:24:23.855604+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
2023-08-16T12:24:23.855604+00:00 app[web.1]: from /app/vendor/ruby-3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.10/libexec/bundle:33:in `<top (required)>'
2023-08-16T12:24:23.855605+00:00 app[web.1]: from /app/bin/bundle:5:in `load'
2023-08-16T12:24:23.855605+00:00 app[web.1]: from /app/bin/bundle:5:in `<main>'
2023-08-16T12:24:24.200300+00:00 heroku[web.1]: Process exited with status 1

Which boils down to:

This is what I get locally when shutting the server down, just to confirm the deprecation warning:

[65598] - Gracefully shutting down workers...
DEPRECATION WARNING: Using a :default format for Time#to_s is deprecated. Please use Time#to_fs instead. (called from <main> at bin/rails:6)
[65598] === puma shutdown: 16/08/2023 13:52:35 ===
[65598] - Goodbye!

To test my theory that it was the timestamp causing the issue I've put up this to test in Heroku, which will use Rails .to_fs if the Time object responds to it, so the deprecation warning goes away and I get this when shutting down:

2023-08-16T12:09:43.351566+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2023-08-16T12:09:43.398164+00:00 app[web.1]: [2] === puma shutdown: 16/08/2023 12:09:43 ===
2023-08-16T12:09:43.401845+00:00 app[web.1]: [2] - Goodbye!
2023-08-16T12:09:43.403954+00:00 app[web.1]: [2] - Gracefully shutting down workers...

So, it looks like removing the deprecation warning will fix my issue unless anyone thinks it is something else?

Puma config:

Config:

# frozen_string_literal: true

# Puma can serve each request in a thread from an internal thread pool.
# The `threads` method setting takes two numbers: a minimum and maximum.
# Any libraries that use thread pools should be configured to match
# the maximum value specified for Puma. Default is set to 5 threads for minimum
# and maximum; this matches the default thread size of Active Record.
#
max_threads_count = ENV.fetch("RAILS_MAX_THREADS", 5)
min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count }
threads min_threads_count, max_threads_count

# Specifies the `port` that Puma will listen on to receive requests; default is 3000.
#
port ENV.fetch("PORT", 3001)

# Specifies the `environment` that Puma will run in.
#
environment ENV.fetch("RAILS_ENV", "development")

# Specifies the `pidfile` that Puma will use.
pidfile ENV.fetch("PIDFILE", "tmp/pids/server.pid")

# Specifies the number of `workers` to boot in clustered mode.
# Workers are forked web server processes. If using threads and workers together
# the concurrency of the application would be max `threads` * `workers`.
# Workers do not work on JRuby or Windows (both of which do not support
# processes).
#
workers ENV.fetch("WEB_CONCURRENCY", 2)

# Use the `preload_app!` method when specifying a `workers` number.
# This directive tells Puma to first boot the application and load code
# before forking the application. This takes advantage of Copy On Write
# process behavior so workers use less memory.
#
preload_app!

# Allow puma to be restarted by `rails restart` command.
plugin :tmp_restart

Start command:

bundle exec puma -C config/puma.rb

Desktop (please complete the following information):

OS: Heroku
Puma Version: 6.3.0 (ruby 3.2.2-p53)
Rails: 7.0.7