http: proxy error: internal error: 101 switching protocols response with non-writable body on macOS Monterey/M1 with Rails 6 and 7
kylebragger opened this issue · comments
With any Rails app when working locally, even a fresh testbed app, I am getting the following error in the puma logs when a connection is attempted to the /cable endpoint for ActionCable:
http: proxy error: internal error: 101 switching protocols response with non-writable body
This does not happen on my Intel mac, only M1, and I've confirmed identical versions of everything -- Rails, all gems, puma-dev, redis, Chrome, etc. The only variable that I can see is M1 vs Intel.
A snippet of the Rails logs:
Started GET "/cable" for 127.0.0.1 at 2022-02-07 11:53:40 -0500
Started GET "/cable/" [WebSocket] for 127.0.0.1 at 2022-02-07 11:53:40 -0500
Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: websocket)
WebSocket error occurred: Broken pipe
Finished "/cable/" [WebSocket] for 127.0.0.1 at 2022-02-07 11:53:40 -0500
Started GET "/cable" for 127.0.0.1 at 2022-02-07 11:53:41 -0500
Started GET "/cable/" [WebSocket] for 127.0.0.1 at 2022-02-07 11:53:41 -0500
Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: websocket)
Finished "/cable/" [WebSocket] for 127.0.0.1 at 2022-02-07 11:53:41 -0500
In the dev console, I see that /cable is returning a 502 response (NB: Safari shows this, while Chrome just shows the request never completed.) I don't see any other errors or exceptions.
Versions:
puma-dev -V
Version: 0.18.1 (go1.17.6)
ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [arm64-darwin21]
rails -v
Rails 6.1.4.4
redis-server -v
Redis server v=6.2.6 sha=00000000:0 malloc=libc bits=64 build=c6f3693d1aced7d9
Gemfile.lock is below:
GEM
remote: https://rubygems.org/
specs:
actioncable (6.1.4.4)
actionpack (= 6.1.4.4)
activesupport (= 6.1.4.4)
nio4r (~> 2.0)
websocket-driver (>= 0.6.1)
actionmailbox (6.1.4.4)
actionpack (= 6.1.4.4)
activejob (= 6.1.4.4)
activerecord (= 6.1.4.4)
activestorage (= 6.1.4.4)
activesupport (= 6.1.4.4)
mail (>= 2.7.1)
actionmailer (6.1.4.4)
actionpack (= 6.1.4.4)
actionview (= 6.1.4.4)
activejob (= 6.1.4.4)
activesupport (= 6.1.4.4)
mail (~> 2.5, >= 2.5.4)
rails-dom-testing (~> 2.0)
actionpack (6.1.4.4)
actionview (= 6.1.4.4)
activesupport (= 6.1.4.4)
rack (~> 2.0, >= 2.0.9)
rack-test (>= 0.6.3)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.0, >= 1.2.0)
actiontext (6.1.4.4)
actionpack (= 6.1.4.4)
activerecord (= 6.1.4.4)
activestorage (= 6.1.4.4)
activesupport (= 6.1.4.4)
nokogiri (>= 1.8.5)
actionview (6.1.4.4)
activesupport (= 6.1.4.4)
builder (~> 3.1)
erubi (~> 1.4)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.1, >= 1.2.0)
activejob (6.1.4.4)
activesupport (= 6.1.4.4)
globalid (>= 0.3.6)
activemodel (6.1.4.4)
activesupport (= 6.1.4.4)
activerecord (6.1.4.4)
activemodel (= 6.1.4.4)
activesupport (= 6.1.4.4)
activestorage (6.1.4.4)
actionpack (= 6.1.4.4)
activejob (= 6.1.4.4)
activerecord (= 6.1.4.4)
activesupport (= 6.1.4.4)
marcel (~> 1.0.0)
mini_mime (>= 1.1.0)
activesupport (6.1.4.4)
concurrent-ruby (~> 1.0, >= 1.0.2)
i18n (>= 1.6, < 2)
minitest (>= 5.1)
tzinfo (~> 2.0)
zeitwerk (~> 2.3)
addressable (2.8.0)
public_suffix (>= 2.0.2, < 5.0)
awesome_print (1.9.2)
base58 (0.2.3)
bindex (0.8.1)
bootsnap (1.10.1)
msgpack (~> 1.2)
builder (3.2.4)
byebug (11.1.3)
capybara (3.36.0)
addressable
matrix
mini_mime (>= 0.1.3)
nokogiri (~> 1.8)
rack (>= 1.6.0)
rack-test (>= 0.6.3)
regexp_parser (>= 1.5, < 3.0)
xpath (~> 3.2)
childprocess (4.1.0)
concurrent-ruby (1.1.9)
crass (1.0.6)
erubi (1.10.0)
eventmachine (1.2.7)
faye-websocket (0.11.1)
eventmachine (>= 0.12.0)
websocket-driver (>= 0.5.1)
ffi (1.15.5)
globalid (1.0.0)
activesupport (>= 5.0)
google-protobuf (3.19.3)
googleapis-common-protos-types (1.3.0)
google-protobuf (~> 3.14)
graphiql-rails (1.8.0)
railties
sprockets-rails
graphql (1.13.6)
grpc (1.43.1)
google-protobuf (~> 3.18)
googleapis-common-protos-types (~> 1.0)
httparty (0.20.0)
mime-types (~> 3.0)
multi_xml (>= 0.5.2)
i18n (1.8.11)
concurrent-ruby (~> 1.0)
jbuilder (2.11.5)
actionview (>= 5.0.0)
activesupport (>= 5.0.0)
listen (3.7.1)
rb-fsevent (~> 0.10, >= 0.10.3)
rb-inotify (~> 0.9, >= 0.9.10)
loofah (2.13.0)
crass (~> 1.0.2)
nokogiri (>= 1.5.9)
mail (2.7.1)
mini_mime (>= 0.1.1)
marcel (1.0.2)
matrix (0.4.2)
method_source (1.0.0)
mime-types (3.4.1)
mime-types-data (~> 3.2015)
mime-types-data (3.2022.0105)
mini_mime (1.1.2)
mini_portile2 (2.7.1)
minitest (5.15.0)
msgpack (1.4.3)
multi_xml (0.6.0)
newrelic-infinite_tracing (8.3.0)
grpc (~> 1.34)
newrelic_rpm (= 8.3.0)
newrelic_rpm (8.3.0)
nio4r (2.5.8)
nokogiri (1.13.1)
mini_portile2 (~> 2.7.0)
racc (~> 1.4)
pg (1.2.3)
phonelib (0.6.55)
public_suffix (4.0.6)
puma (5.5.2)
nio4r (~> 2.0)
racc (1.6.0)
rack (2.2.3)
rack-cors (1.1.1)
rack (>= 2.0.0)
rack-mini-profiler (2.3.3)
rack (>= 1.2.0)
rack-proxy (0.7.2)
rack
rack-test (1.1.0)
rack (>= 1.0, < 3)
rails (6.1.4.4)
actioncable (= 6.1.4.4)
actionmailbox (= 6.1.4.4)
actionmailer (= 6.1.4.4)
actionpack (= 6.1.4.4)
actiontext (= 6.1.4.4)
actionview (= 6.1.4.4)
activejob (= 6.1.4.4)
activemodel (= 6.1.4.4)
activerecord (= 6.1.4.4)
activestorage (= 6.1.4.4)
activesupport (= 6.1.4.4)
bundler (>= 1.15.0)
railties (= 6.1.4.4)
sprockets-rails (>= 2.0.0)
rails-dom-testing (2.0.3)
activesupport (>= 4.2.0)
nokogiri (>= 1.6)
rails-html-sanitizer (1.4.2)
loofah (~> 2.3)
railties (6.1.4.4)
actionpack (= 6.1.4.4)
activesupport (= 6.1.4.4)
method_source
rake (>= 0.13)
thor (~> 1.0)
rake (13.0.6)
rb-fsevent (0.11.0)
rb-inotify (0.10.1)
ffi (~> 1.0)
redis (4.5.1)
regexp_parser (2.2.0)
rexml (3.2.5)
rubyzip (2.3.2)
sass-rails (6.0.0)
sassc-rails (~> 2.1, >= 2.1.1)
sassc (2.4.0)
ffi (~> 1.9)
sassc-rails (2.1.2)
railties (>= 4.0.0)
sassc (>= 2.0)
sprockets (> 3.0)
sprockets-rails
tilt
selenium-webdriver (4.1.0)
childprocess (>= 0.5, < 5.0)
rexml (~> 3.2, >= 3.2.5)
rubyzip (>= 1.2.2)
semantic_range (3.0.0)
sentry-rails (5.0.0)
railties (>= 5.0)
sentry-ruby-core (~> 5.0.0)
sentry-ruby (5.0.0)
concurrent-ruby (~> 1.0, >= 1.0.2)
sentry-ruby-core (= 5.0.0)
sentry-ruby-core (5.0.0)
concurrent-ruby
skylight (5.1.1)
activesupport (>= 5.2.0)
solana_rpc_ruby (1.2.0)
faye-websocket (~> 0.11)
spring (4.0.0)
sprockets (4.0.2)
concurrent-ruby (~> 1.0)
rack (> 1, < 3)
sprockets-rails (3.4.2)
actionpack (>= 5.2)
activesupport (>= 5.2)
sprockets (>= 3.0.0)
thor (1.2.1)
tilt (2.0.10)
turbolinks (5.2.1)
turbolinks-source (~> 5.2)
turbolinks-source (5.2.0)
tzinfo (2.0.4)
concurrent-ruby (~> 1.0)
web-console (4.2.0)
actionview (>= 6.0.0)
activemodel (>= 6.0.0)
bindex (>= 0.4.0)
railties (>= 6.0.0)
webdrivers (5.0.0)
nokogiri (~> 1.6)
rubyzip (>= 1.3.0)
selenium-webdriver (~> 4.0)
webpacker (5.4.3)
activesupport (>= 5.2)
rack-proxy (>= 0.6.1)
railties (>= 5.2)
semantic_range (>= 2.3.0)
websocket-driver (0.7.5)
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.5)
xpath (3.2.0)
nokogiri (~> 1.8)
zeitwerk (2.5.3)
PLATFORMS
ruby
DEPENDENCIES
awesome_print
base58
bootsnap (>= 1.4.4)
byebug
capybara (>= 3.26)
graphiql-rails
graphql
httparty
jbuilder (~> 2.7)
listen (~> 3.3)
newrelic-infinite_tracing
newrelic_rpm
pg (~> 1.1)
phonelib
puma (~> 5.0)
rack-cors
rack-mini-profiler (~> 2.0)
rails (~> 6.1.4, >= 6.1.4.1)
redis (~> 4.0)
sass-rails (>= 6)
selenium-webdriver
sentry-rails
sentry-ruby
skylight
solana_rpc_ruby
spring
turbolinks (~> 5)
tzinfo-data
web-console (>= 4.1.0)
webdrivers
webpacker (~> 5.0)
RUBY VERSION
ruby 2.7.2p137
BUNDLED WITH
2.1.4
Are you able to confirm whether this is an issue with v0.17.0 ? I suspect this is a regression introduced in #292.
After doing a little sleuthing, I think we'll need to re-apply some patches that were previously tucked away inside the vendored httputil, that patched in support for websockets before go supported them in 1.12+.
@nonrational thanks for the speedy reply! I just grabbed 0.17 and that seemed to do the trick.
@nonrational I can take a look since I broke this. I'm also on Monterey on M1 Pro
I was able to reproduce the problem when configuring puma-dev to proxy connections to a WebSocket server.
WireShark shows that puma-dev responds erroneously with 502 Bad Gateway when receiving the WebSocket protocol handshake.
The connection from puma-dev to the backend server is actually fine:
The issue is resolved in #304. I had been working on that branch as part of #290 already. I split out the two PRs because I thought they could be released independently. What I didn't realize was that the logic for proxying WebSocket connections is implemented in http.Transport in the modern stdlib. So I ended up removing WebSocket support from the ReverseProxy in #292, without restoring functionality in the Transport. Upgrading to stdlib for both http and httputil fixes the problem and we just let the stdlib do the heavy lifting.
Amazing work. Thanks @cjlarose! 🙏
@kylebragger are you able to grab the binary from https://github.com/puma/puma-dev/releases/tag/v0.18.2 and verify on your end?
@nonrational yep, will check it out soon and report back. thanks for the quick work! /cc @cjlarose
I had the same issue with puma-dev 0.18.1, but on Intel (iMac 2014, macOS Big Sur 11.6.3) with Ruby 3.1.0. After installing the binary of 0.18.2 (overriding the existing one from Homebrew), it all works fine again 👍
Thanks @ledermann. v0.18.2 is now available via homebrew.