Puma threads deadlock when multiple simultaneous requests

Question

Puma threads deadlock when multiple simultaneous requests

saudraisn opened this issue 2 years ago · comments

Describe the bug
When launching multiple queries against our rails server simulteanously (multiple React-Query queries firing in our React frontend application), we are experimenting a deadlock in puma threads, which jams our app server.

This interruption happens even before any of our rails app code is reached. As you will see in the logs below, the threads appear to lock while loading controller classes (controller_class_for method call). I must also mention that no error happens while running in single threaded mode (threads 1,1), which makes me think that our routes/controllers setup is built correctly, since rails can load them perfectly in a single thread. It also happens just after a fresh startup of the backend server.

When troubleshooting this issue, I tried the ActionDispatch::DebugLocks method to get some information of what's going on in these threads and here's the results:

Thread 0 [0x102d4 sleep]  Sharing

/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/inflector/methods.rb:280:in `const_get'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/inflector/methods.rb:280:in `constantize'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/core_ext/string/inflections.rb:74:in `constantize'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/http/request.rb:90:in `controller_class_for'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/request/utils.rb:95:in `action_encoding_template'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/request/utils.rb:83:in `encode'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/request/utils.rb:45:in `set_binary_encoding'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/http/parameters.rb:68:in `path_parameters='
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/journey/router.rb:48:in `block in serve'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/routing/route_set.rb:852:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/conditional_get.rb:40:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
/bundle/vendor/ruby/3.1.0/gems/activerecord-7.0.3.1/lib/active_record/migration.rb:603:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/callbacks.rb:99:in `run_callbacks'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
/bundle/vendor/ruby/3.1.0/gems/sentry-rails-4.9.0/lib/sentry/rails/rescued_exception_interceptor.rb:9:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
/bundle/vendor/ruby/3.1.0/gems/sentry-ruby-core-4.9.0/lib/sentry/rack/capture_exceptions.rb:11:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:26:in `call'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/rack/logger.rb:40:in `call_app'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/rack/logger.rb:25:in `block in call'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/tagged_logging.rb:114:in `block in tagged'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/tagged_logging.rb:38:in `tagged'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/tagged_logging.rb:114:in `tagged'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/rack/logger.rb:25:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/server_timing.rb:20:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/static.rb:23:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/debug_locks.rb:41:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:137:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-cors-1.1.1/lib/rack/cors.rb:100:in `call'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/engine.rb:530:in `call'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/configuration.rb:252:in `call'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/request.rb:77:in `block in handle_request'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/request.rb:76:in `handle_request'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/server.rb:441:in `process_client'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/thread_pool.rb:147:in `block in spawn_thread'


---


Thread 1 [0x102e8 sleep]  Sharing

/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/inflector/methods.rb:280:in `const_get'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/inflector/methods.rb:280:in `constantize'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/core_ext/string/inflections.rb:74:in `constantize'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/http/request.rb:90:in `controller_class_for'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/request/utils.rb:95:in `action_encoding_template'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/request/utils.rb:83:in `encode'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/request/utils.rb:45:in `set_binary_encoding'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/http/parameters.rb:68:in `path_parameters='
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/journey/router.rb:48:in `block in serve'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/routing/route_set.rb:852:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/conditional_get.rb:40:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
/bundle/vendor/ruby/3.1.0/gems/activerecord-7.0.3.1/lib/active_record/migration.rb:603:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/callbacks.rb:99:in `run_callbacks'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
/bundle/vendor/ruby/3.1.0/gems/sentry-rails-4.9.0/lib/sentry/rails/rescued_exception_interceptor.rb:9:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
/bundle/vendor/ruby/3.1.0/gems/sentry-ruby-core-4.9.0/lib/sentry/rack/capture_exceptions.rb:11:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:26:in `call'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/rack/logger.rb:40:in `call_app'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/rack/logger.rb:25:in `block in call'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/tagged_logging.rb:114:in `block in tagged'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/tagged_logging.rb:38:in `tagged'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/tagged_logging.rb:114:in `tagged'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/rack/logger.rb:25:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/server_timing.rb:20:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/static.rb:23:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/debug_locks.rb:41:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:137:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-cors-1.1.1/lib/rack/cors.rb:100:in `call'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/engine.rb:530:in `call'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/configuration.rb:252:in `call'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/request.rb:77:in `block in handle_request'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/request.rb:76:in `handle_request'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/server.rb:441:in `process_client'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/thread_pool.rb:147:in `block in spawn_thread'


---


Thread 2 [0x102c0 sleep]  Sharing

/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_controller/railtie.rb:81:in `block (3 levels) in <class:Railtie>'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_controller/railtie.rb:77:in `each'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_controller/railtie.rb:77:in `block (2 levels) in <class:Railtie>'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/lazy_load_hooks.rb:82:in `class_eval'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/lazy_load_hooks.rb:82:in `block in execute_hook'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/lazy_load_hooks.rb:72:in `with_execution_control'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/lazy_load_hooks.rb:77:in `execute_hook'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/lazy_load_hooks.rb:63:in `block in run_load_hooks'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/lazy_load_hooks.rb:62:in `each'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/lazy_load_hooks.rb:62:in `run_load_hooks'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_controller/api.rb:148:in `<class:API>'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_controller/api.rb:89:in `<module:ActionController>'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_controller/api.rb:7:in `<main>'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/loaded_features_index.rb:88:in `register'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
/bundle/vendor/ruby/3.1.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:35:in `require'
/workspace/packages/server/app/controllers/application_controller.rb:4:in `<main>'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/loaded_features_index.rb:88:in `register'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
/bundle/vendor/ruby/3.1.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:27:in `require'
/workspace/packages/server/app/controllers/graphql_controller.rb:2:in `<main>'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/loaded_features_index.rb:88:in `register'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
/bundle/vendor/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
/bundle/vendor/ruby/3.1.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:27:in `require'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/inflector/methods.rb:280:in `const_get'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/inflector/methods.rb:280:in `constantize'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/core_ext/string/inflections.rb:74:in `constantize'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/http/request.rb:90:in `controller_class_for'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/request/utils.rb:95:in `action_encoding_template'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/request/utils.rb:83:in `encode'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/request/utils.rb:45:in `set_binary_encoding'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/http/parameters.rb:68:in `path_parameters='
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/journey/router.rb:48:in `block in serve'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/routing/route_set.rb:852:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/conditional_get.rb:40:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
/bundle/vendor/ruby/3.1.0/gems/activerecord-7.0.3.1/lib/active_record/migration.rb:603:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/callbacks.rb:99:in `run_callbacks'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
/bundle/vendor/ruby/3.1.0/gems/sentry-rails-4.9.0/lib/sentry/rails/rescued_exception_interceptor.rb:9:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
/bundle/vendor/ruby/3.1.0/gems/sentry-ruby-core-4.9.0/lib/sentry/rack/capture_exceptions.rb:11:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:26:in `call'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/rack/logger.rb:40:in `call_app'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/rack/logger.rb:25:in `block in call'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/tagged_logging.rb:114:in `block in tagged'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/tagged_logging.rb:38:in `tagged'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/tagged_logging.rb:114:in `tagged'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/rack/logger.rb:25:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
/bundle/vendor/ruby/3.1.0/gems/activesupport-7.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/server_timing.rb:20:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/static.rb:23:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/debug_locks.rb:41:in `call'
/bundle/vendor/ruby/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:137:in `call'
/bundle/vendor/ruby/3.1.0/gems/rack-cors-1.1.1/lib/rack/cors.rb:100:in `call'
/bundle/vendor/ruby/3.1.0/gems/railties-7.0.3.1/lib/rails/engine.rb:530:in `call'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/configuration.rb:252:in `call'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/request.rb:77:in `block in handle_request'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/request.rb:76:in `handle_request'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/server.rb:441:in `process_client'
/bundle/vendor/ruby/3.1.0/gems/puma-5.6.4/lib/puma/thread_pool.rb:147:in `block in spawn_thread'

Puma config:

# Puma can serve each request in a thread from an internal thread pool.
# The `threads` method setting takes two numbers: a minimum and maximum.
# Any libraries that use thread pools should be configured to match
# the maximum value specified for Puma. Default is set to 5 threads for minimum
# and maximum; this matches the default thread size of Active Record.
#
max_threads_count = ENV.fetch('RAILS_MAX_THREADS') { 10 }
min_threads_count = ENV.fetch('RAILS_MIN_THREADS') { max_threads_count }
threads min_threads_count, max_threads_count

# Specifies the `worker_timeout` threshold that Puma will use to wait before
# terminating a worker in development environments.
#
worker_timeout 3600 if ENV.fetch('RAILS_ENV', 'development') == 'development'

# Specifies the `port` that Puma will listen on to receive requests; default is 3000.
#
port ENV.fetch('PORT') { 3000 }

# Specifies the `environment` that Puma will run in.
#
environment ENV.fetch('RAILS_ENV') { 'development' }

# Specifies the `pidfile` that Puma will use.
pidfile ENV.fetch('PIDFILE') { 'tmp/pids/server.pid' }

# Specifies the number of `workers` to boot in clustered mode.
# Workers are forked web server processes. If using threads and workers together
# the concurrency of the application would be max `threads` * `workers`.
# Workers do not work on JRuby or Windows (both of which do not support
# processes).
#
workers ENV.fetch('WEB_CONCURRENCY') { 4 } if ENV['RENDER'] == 'true'

# Use the `preload_app!` method when specifying a `workers` number.
# This directive tells Puma to first boot the application and load code
# before forking the application. This takes advantage of Copy On Write
# process behavior so workers use less memory.
#
preload_app! if ENV['RENDER'] == 'true'

# Allow puma to be restarted by `rails restart` command.
plugin :tmp_restart

rails server -b 0.0.0.0 -p 4000

To Reproduce

I will be more than happy to schedule some time with anyone willing to help to show the app setup that consistently recreates that race condition.

Expected behavior

The threads are expected to execute the requests and not fall in a deadlock condition

Desktop (please complete the following information):

=> Booting Puma
=> Rails 7.0.3.1 application starting in development 
=> Run `bin/rails server --help` for more startup options
Puma starting in single mode...
* Puma version: 5.6.4 (ruby 3.1.2-p20) ("Birdie's Version")
*  Min threads: 10
*  Max threads: 10
*  Environment: development
*          PID: 4085
* Listening on http://0.0.0.0:4000
Use Ctrl-C to stop

OS: (running inside a vscode devcontainer on macOS Monterey 12.4, M1 chip)

Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

Thanks a lot in advance for your help and this amazing lib

Patrik Ragnarsson · Answer 1 · Tue Aug 16 2022 21:20:46 GMT+0800 (China Standard Time)

This interruption happens even before any of our rails app code is reached. As you will see in the logs below, the threads appear to lock while loading controller classes (controller_class_for method call). I must also mention that no error happens while running in single threaded mode (threads 1,1), which makes me think that our routes/controllers setup is built correctly, since rails can load them perfectly in a single thread. It also happens just after a fresh startup of the backend server.

Sounds like you have already almost figured this one out :)

It does not sound like an issue with Puma.

Sounds like an issue with class loading in Ruby (two threads stuck on Object.const_get). Found this (very old) comment that sounds a bit relevant: https://bugs.ruby-lang.org/issues/8570#note-2)

I think you should report this issue to Rails, but I also think you need to show code how to reproduce the issue (not just offer to walk through your app with someone). Maybe you can find some inspiration from #2655 (there's an example app linked in the comments).

(There's also the INFO signal to dump backtraces of the threads in Puma, but it looks like DebugLocks did the job of that)

Nicolas Saudrais · Answer 2 · Tue Aug 16 2022 21:39:27 GMT+0800 (China Standard Time)

Thanks for the quick response 🙌 , I was not sure wether I should file this issue here or in the rails repo.
And the discussion you linked seems to be exactly the issue that we're experimenting.
I will try to replicate it with a smaller repo before submitting. Thanks again.

Cameron Steffen · Answer 3 · Sat Apr 08 2023 06:21:52 GMT+0800 (China Standard Time)

@saudraisn Did you ever figure this out? I'm experiencing a very similar issue.

Nicolas Saudrais · Answer 4 · Tue Apr 11 2023 20:37:16 GMT+0800 (China Standard Time)

@camsteffen Not for the moment unfortunately. What we ended up doing is running our development server with one thread when doing several backend changes (no deadlock happening). This is a temporary solution and the issue is still in our backlog for us to tackle it later. I'll definitely refer the fix in this thread if anything comes up and might be useful to you! Cheers!

Daniel O · Answer 5 · Mon Apr 24 2023 00:07:11 GMT+0800 (China Standard Time)

We experienced a similar issue (CI pipeline, vue js component kicking off ajax requests). Symptoms were reduced with eager loading, annoyingly the rails debugging approach mentioned above didn't help.

Occasionally, we'd see loading errors like:

XYZ::SOAP::GetDocumentList, expected app/services/xuz/soap/get_document_list.rb to define it (LoadError)

Suggesting the file was marked as 'loaded' but the class was still in the process of being defined in some other request.

https://gist.github.com/phil-monroe/237328ddf2fb4e509dd3ef9e5fca33b7 at least provided a bit of visibility, so a combination of the two approaches might help.

Additionally, check you are using the right autoloader - https://guides.rubyonrails.org/classic_to_zeitwerk_howto.html#thread-safety-everywhere