fails to intercept oauth2

[Peter W]

Hello 👋

I'm trying to intercept some simple ruby code, like this:

# example.rb
require 'oauth2'

puts "env rubylib is:"
puts ENV['RUBYLIB']

client = OAuth2::Client.new(client_id, client_secret, site)

Using a fresh terminal, I run it and it looks like net/https isn't overridden:

env rubylib is:
/Applications/HTTP Toolkit.app/Contents/Resources/app/httptoolkit-server/overrides/gems
Traceback (most recent call last):
	18: from ./example.rb:19:in `<main>'
	17: from /Users/user/.gem/ruby/2.6.0/gems/oauth2-1.4.4/lib/oauth2/strategy/client_credentials.rb:20:in `get_token'
	16: from /Users/user/.gem/ruby/2.6.0/gems/oauth2-1.4.4/lib/oauth2/client.rb:147:in `get_token'
	15: from /Users/user/.gem/ruby/2.6.0/gems/oauth2-1.4.4/lib/oauth2/client.rb:99:in `request'
	14: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/connection.rb:492:in `run_request'
	13: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/rack_builder.rb:153:in `build_response'
	12: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/request/url_encoded.rb:23:in `call'
	11: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter/net_http.rb:66:in `call'
	10: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter.rb:60:in `connection'
	 9: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter/net_http.rb:68:in `block in call'
	 8: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter/net_http.rb:126:in `perform_request'
	 7: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter/net_http.rb:135:in `request_with_wrapped_block'
	 6: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter/net_http.rb:149:in `request_via_request_method'
	 5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:1470:in `request'
	 4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:919:in `start'
	 3: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
	 2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:996:in `connect'
	 1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/protocol.rb:44:in `ssl_socket_connect'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)

(Note the error):
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)

From faraday/adapter/net_http.rb, it looks like it's looking for net/https:

begin
  require 'net/https'
rescue LoadError
  warn 'Warning: no such file to load -- net/https. ' \
    'Make sure openssl is installed if you want ssl support'
  require 'net/http'
end

Should the overrides include a net/https file?

❯ ls "/Applications/HTTP Toolkit.app/Contents/Resources/app/httptoolkit-server/overrides/gems"
http.rb   net       stripe.rb uri
❯ ls "/Applications/HTTP Toolkit.app/Contents/Resources/app/httptoolkit-server/overrides/gems/net"
http.rb

Thanks

[Peter W]

Just an update to confirm a workaround:

Modify faraday/adapter/net_http.rb:

begin
-  require 'net/https'
+  require 'net/http'
rescue LoadError
  warn 'Warning: no such file to load -- net/https. ' \
    'Make sure openssl is installed if you want ssl support'
  require 'net/http'
end

[Tim Perry]

Hmm, in theory overriding net/https shouldn't be required, and just net/http should be sufficient, because net/https doesn't actually do anything: https://github.com/ruby/ruby/blob/master/lib/net/https.rb. net/http includes all the HTTPS logic required all by itself nowadays, so that's where we hook the request setup.

I've just tested and running the below in irb in an intercepted terminal seems to work fine for me:

require 'net/https'
Net::HTTP.get(URI('https://example.com/'))

That runs successfully, and shows up correctly in the HTTP Toolkit UI. If you test out that minimal case, does that work for you? If not, can you let me know which version of Ruby you're using?

If that does work for you, I suspect this is something to do with Faraday or the rest of your setup. Still a bug either way of course! If you can give me a minimal example and Faraday version I'll look into it further.

[Peter W]

Thanks @pimterry.

No luck for me. Is it possible the require_relative 'http' in Ruby's net/https.rb is getting the local source rather than that from HTTP Toolkit?

Here's the version:

❯ ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin19]
❯ which ruby
/usr/bin/ruby

And here's what I see with the minimal example:

❯ irb

WARNING: This version of ruby is included in macOS for compatibility with legacy software.
In future versions of macOS the ruby runtime will not be available by
default, and may require you to install an additional package.

irb(main):001:0> require 'net/https'
=> true
irb(main):002:0> Net::HTTP.get(URI('https://example.com'))
Traceback (most recent call last):
       12: from /usr/bin/irb:23:in `<main>'
       11: from /usr/bin/irb:23:in `load'
       10: from /Library/Ruby/Gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
        9: from (irb):2
        8: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:458:in `get'
        7: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:481:in `get_response'
        6: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:605:in `start'
        5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:919:in `start'
        4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
        3: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:996:in `connect'
        2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/protocol.rb:44:in `ssl_socket_connect'
        1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/protocol.rb:44:in `connect_nonblock'
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain))

[Tim Perry]

I'm back from holidays now, looking into this again, and I've managed to reproduce it! It seems to appear just when using the built-in Ruby release on a Mac. I think the issue is:

• As you suggested, net/https avoids the net/http hook, by using require_relative. If you only use net/https and never require net/http at all then the hook is never used.
• The default Ruby on Mac uses a different OpenSSL build, I think primarily for integration with the system keychain, which ignores OpenSSL's standard SSL_CERT_FILE env var.

If you import net/http too, then the hook handles certificate setup automatically. If you import just net/https in most environments (except a default Mac Ruby install) it still works anyway, which hid this, because the certificate is trusted automatically by OpenSSL itself due to SSL_CERT_FILE. If both issues combine though, then you're in trouble.

As an immediate workaround, adding require 'net/http' before running any code that runs require 'net/https' does seem to resolve the issue, and should have no downside otherwise. Alternatively, if you install and use Ruby via rbenv on Mac (or with rvm or brew, I suspect) then it works fine out of the box.

To fix this more permanently, I've now added a hook for net/https too, which just ensures the net/http hook always runs first, before loading the real module. That seems to fix the issue in my case.

Would you mind giving this a quick test, to confirm it fixes the issue for you? You just need to drop the new https.rb file into HTTP Toolkit's overrides/gems/net folder on your machine. I'm intending to get a release out in the next few days, once I'm confident that this works.

[Tim Perry]

Hi @techieshark - just to let you know, this fix is now released in server 1.1.0. Next time you open HTTP Toolkit it'll update in the background, and the following time everything should work with Ruby and net/https as expected. Thanks so much for reporting the issue, it's great to be able to catch more tricky Ruby interception cases like this!