Request logging on Puma with an empty request body raises a NoMethodError

[Brendan Weibrecht]

We uncovered a bug which had lain dormant in deX Core for a long time; it revealed itself this afternoon when we upgraded Envoy. Requests began failing from GreenLog::Rack::RequestLogging with a NoMethodError.

When there is no request body and no Content-Length: 0 header, Puma suppplies the request body as a Puma::NullIO object. This lacks #pos and #seek, so GreenLog::Rack::RequestLogging#request_body was raising a NoMethodError from both its method body and ensure.

NoMethodError: undefined method `pos' for #<Puma::NullIO:0x00005652a142d7a0>
  from green_log/rack/request_logging.rb:67:in `request_body'
  from green_log/rack/request_logging.rb:62:in `optional_request_details'
  from green_log/rack/request_logging.rb:55:in `request_details'
  from green_log/rack/request_logging.rb:41:in `log'
  from green_log/rack/request_logging.rb:25:in `call'
  from sinatra/base.rb:986:in `forward'
  from sinatra/base.rb:1078:in `route_missing'
  from sinatra/base.rb:1027:in `route!'
  from sinatra/base.rb:1023:in `route!'
  from sinatra/base.rb:1129:in `block in dispatch!'
  from sinatra/base.rb:1101:in `block in invoke'
  from sinatra/base.rb:1101:in `catch'
  from sinatra/base.rb:1101:in `invoke'
  from sinatra/base.rb:1124:in `dispatch!'
  from sinatra/base.rb:939:in `block in call!'
  from sinatra/base.rb:1101:in `block in invoke'
  from sinatra/base.rb:1101:in `catch'
  from sinatra/base.rb:1101:in `invoke'
  from sinatra/base.rb:939:in `call!'
  from sinatra/base.rb:929:in `call'
  from ddtrace/contrib/sinatra/tracer_middleware.rb:35:in `block in call'
  from ddtrace/tracer.rb:307:in `trace'
  from ddtrace/contrib/sinatra/tracer_middleware.rb:23:in `call'
  from rack/protection/xss_header.rb:18:in `call'
  from rack/protection/path_traversal.rb:16:in `call'
  from rack/protection/json_csrf.rb:26:in `call'
  from rack/protection/base.rb:50:in `call'
  from rack/protection/base.rb:50:in `call'
  from rack/protection/frame_options.rb:31:in `call'
  from rack/null_logger.rb:11:in `call'
  from rack/head.rb:12:in `call'
  from sinatra/base.rb:216:in `call'
  from sinatra/base.rb:1991:in `call'
  from lib/dex/core/api/server_error/internal_error_handler.rb:17:in `call'
  from raven/integrations/rack.rb:51:in `call'
  from ddtrace/contrib/rack/middlewares.rb:85:in `call'
  from puma/configuration.rb:249:in `call'
  from puma/request.rb:76:in `block in handle_request'
  from puma/thread_pool.rb:338:in `with_force_shutdown'
  from puma/request.rb:75:in `handle_request'
  from puma/server.rb:437:in `process_client'
  from puma/thread_pool.rb:145:in `block in spawn_thread'
NoMethodError: undefined method `seek' for #<Puma::NullIO:0x00005652a142d7a0>
  from green_log/rack/request_logging.rb:71:in `ensure in request_body'
  from green_log/rack/request_logging.rb:71:in `request_body'
  from green_log/rack/request_logging.rb:62:in `optional_request_details'
  from green_log/rack/request_logging.rb:55:in `request_details'
  from green_log/rack/request_logging.rb:41:in `log'
  from green_log/rack/request_logging.rb:25:in `call'
  from sinatra/base.rb:986:in `forward'
  from sinatra/base.rb:1078:in `route_missing'
  from sinatra/base.rb:1027:in `route!'
  from sinatra/base.rb:1023:in `route!'
  from sinatra/base.rb:1129:in `block in dispatch!'
  from sinatra/base.rb:1101:in `block in invoke'
  from sinatra/base.rb:1101:in `catch'
  from sinatra/base.rb:1101:in `invoke'
  from sinatra/base.rb:1124:in `dispatch!'
  from sinatra/base.rb:939:in `block in call!'
  from sinatra/base.rb:1101:in `block in invoke'
  from sinatra/base.rb:1101:in `catch'
  from sinatra/base.rb:1101:in `invoke'
  from sinatra/base.rb:939:in `call!'
  from sinatra/base.rb:929:in `call'
  from ddtrace/contrib/sinatra/tracer_middleware.rb:35:in `block in call'
  from ddtrace/tracer.rb:307:in `trace'
  from ddtrace/contrib/sinatra/tracer_middleware.rb:23:in `call'
  from rack/protection/xss_header.rb:18:in `call'
  from rack/protection/path_traversal.rb:16:in `call'
  from rack/protection/json_csrf.rb:26:in `call'
  from rack/protection/base.rb:50:in `call'
  from rack/protection/base.rb:50:in `call'
  from rack/protection/frame_options.rb:31:in `call'
  from rack/null_logger.rb:11:in `call'
  from rack/head.rb:12:in `call'
  from sinatra/base.rb:216:in `call'
  from sinatra/base.rb:1991:in `call'
  from lib/dex/core/api/server_error/internal_error_handler.rb:17:in `call'
  from raven/integrations/rack.rb:51:in `call'
  from ddtrace/contrib/rack/middlewares.rb:85:in `call'
  from puma/configuration.rb:249:in `call'
  from puma/request.rb:76:in `block in handle_request'
  from puma/thread_pool.rb:338:in `with_force_shutdown'
  from puma/request.rb:75:in `handle_request'
  from puma/server.rb:437:in `process_client'
  from puma/thread_pool.rb:145:in `block in spawn_thread'

@nickburgin managed to track down the cause:

From the Envoy changelog of 1.18.0:

http: no longer adding content-length: 0 for requests which should not have bodies. This behavior can be temporarily reverted by setting envoy.reloadable_features.dont_add_content_length_for_bodiless_requests false.

We don't log request bodies locally, so hadn't realised this bug was only being mitigated by an unusual behaviour of our ingress. We proved that this was the culprit by manually adding this header to a curl locally, which made requests to our application work.

We've created a patch, so will submit a PR.