gRPC doesn't respect the no_proxy environment variable
shabzy opened this issue · comments
Issue found using the helloworld greeter server/client example, but should apply to everyone. Using git clone from today.
If on a linux machine, the environment has the usual "http_proxy" environment variable configured, gRPC will take that into account when trying to connect, however, will then proceed to ignore the companion no_proxy setting:
For example:
$ env
http_proxy=http://106.1.216.121:8080
no_proxy=localhost,127.0.0.1
$ ./greeter_client
D0306 16:00:11.419586349 1897 combiner.c:351] C:0x25a9290 finish old_state=3
D0306 16:00:11.420527744 1896 tcp_client_posix.c:179] CLIENT_CONNECT: ipv4:106.1.216.121:8080: on_writable: error="No Error"
D0306 16:00:11.420567382 1896 combiner.c:145] C:0x25a69a0 create
D0306 16:00:11.420581887 1896 tcp_client_posix.c:119] CLIENT_CONNECT: ipv4:106.1.216.121:8080: on_alarm: error="Cancelled"
I0306 16:00:11.420617663 1896 http_connect_handshaker.c:319] Connecting to server 127.0.0.1:50051 via HTTP proxy ipv4:106.1.216.121:8080
Basically, it's using the http_proxy url to connect even though localhost is in the no_proxy list. Since the default for no_proxy includes localhost on most linux machines; the end result is that any user with an http_proxy configured will never be able to connect to localhost.
Using the ruby client (1.2.2, ruby 2.3.0) the error output is a much more sparse than the C++ logging, making it difficult to identify the issues is with a proxy:
[root]# /opt/ruby/2.3/bin/bundle exec ./greeter_client.rb
bundler: failed to load command: ./greeter_client.rb (./greeter_client.rb)
GRPC::Unavailable: 14:Connect Failed
/opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:46:in `check_status'
/opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:178:in `attach_status_results_and_complete_call'
/opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:340:in `request_response'
/opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:167:in `request_response'
/opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:185:in `block (3 levels) in rpc_stub_class'
/root/grpc/examples/ruby/greeter_client.rb:46:in `main'
/root/grpc/examples/ruby/greeter_client.rb:50:in `<top (required)>'
Adding the environment variables GRPC_VERBOSITY=DEBUG GRPC_TRACE=all produced more information, which was very helpful in debugging:
[
{
"created": "@1491939664.942593641",
"description": "HTTP proxy returned response code 502",
"file": "src/core/ext/client_channel/http_connect_handshaker.c",
"file_line": 229
}
]In python, similar case:
[root]# python3 greeter_client.py
Traceback (most recent call last):
File "greeter_client.py", line 48, in <module>
run()
File "greeter_client.py", line 43, in run
response = stub.SayHello(helloworld_pb2.HelloRequest(name='you'))
File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 507, in __call__
return _end_unary_response_blocking(state, call, False, deadline)
File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 455, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>```
It's unclear if "connect" in "connect failed" is HTTP CONNECT. If the status was more expressive, say "Proxy failed to connect to host" or some other details, this would have been more easily debuggable. I expect this would be equally hard to debug in C#.
I just encountered this same problem using gRPC v1.2.2 C#. I had HTTP_PROXY and HTTPS_PROXY set on my windows system and the greeter example in C# was failing to connect. Removing the env variables fixed the issue, but that's not a viable solution.
I tried adding a NO_PROXY env variable but it had no effect. What's further strange is that the client doesn't fail to connect, the connect itself succeeds. However, the first call to an API endpoint fails with StatusCode=Unavailable and Detail="Endpoint read failed".
Another interesting data point, this behavior does not happen using gRPC v1.0.1. This may because gRPC did not support proxy env variables in that version, hard to say for sure.
Confirmed that this is still a problem today.
in case you still experience the issue please run your program with setting to empty string the http_proxy string:
HTTP_PROXY= http_proxy= ./greeter_client
@stefanofiorentino do you still have this issue with the latest master branch code? Do you have localhost in your no_proxy env var?
Not fixed yet:
Windows 10 x64
grpcio 1.4.0
python 3.6.2
http_proxy=http://some_proxy:111
https_proxy=https://other_proxy:111
no_proxy=localhost,127.0.0.1
Same error above.
Thanks!.
The change hasn't made its way into a release yet. It will be in the upcoming 1.6 release.
no_proxy variable is still being ignored in version v1.13.0.
@dhrsharma I just checked with v1.13.0, and it seems to be working. Can you share your no_proxy, http_proxy environment variables and the service url you're trying to use?
@bsyk my mistake. Yes, it works with v1.13.0. Actually, I was not creating a new client after changing the no_proxy at runtime. Sorry for the trouble.