Error receiving from doppler via gRPC context canceled

[ashwithahn]

Hi Team,
I am unable to retrieve Metrics and Logs. When i do cf autoscaling-metrics bigapp2 memoryused i received :

Showing metrics for app bigapp2...
OK

No metrics found for app bigapp2.

cf nozzle --debug gives the below error

WEBSOCKET RESPONSE
HTTP/1.1 400 Bad Request
X-Vcap-Request-Id: ff57d685-c478-4ff5-4d86-b92337e14afd
Content-Length: 12
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Tue, 17 Jul 2018 06:34:41 GMT
Sec-Websocket-Version: 13
X-Content-Type-Options: nosniff
Error dialing trafficcontroller server: websocket: bad handshake.
Please ask your Cloud Foundry Operator to check the platform configuration (trafficcontroller is wss://doppler.app-cloudfoundry.com:443).

Further investigating this, found that in log-api -> loggregator_trafficcontroller -> trafficcontroller.log

Startup: Setting up the loggregator traffic controller
2018/07/17 03:10:09 health bound to: 127.0.0.1:14825
2018/07/17 03:10:09 Metrics endpoint is listening on 127.0.0.1:14825
2018/07/17 03:10:09 Event from finder: {GRPCDopplers:[10.0.16.16:8082]}
2018/07/17 03:10:09 adding doppler 10.0.16.16:8082
2018/07/17 03:10:09 pprof bound to: 127.0.0.1:41142
2018/07/17 03:10:09 ws bound to: 0.0.0.0:8081
2018/07/17 04:08:36 Connecting to 1 dopplers
2018/07/17 04:08:36 websocket handler: Not a websocket handshake: websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
2018/07/17 04:08:36 Disconnecting from stream (10.0.16.16:8082) (doppler.disconnect=false) (ctx.disconnect=1)
2018/07/17 04:08:36 error receiving from doppler via gRPC context canceled
2018/07/17 04:08:37 Connecting to 1 dopplers
2018/07/17 04:08:37 websocket handler: Not a websocket handshake: websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
2018/07/17 04:08:37 Disconnecting from stream (10.0.16.16:8082) (doppler.disconnect=false) (ctx.disconnect=1)
2018/07/17 04:08:37 error receiving from doppler via gRPC context canceled

Please help

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/159355288

The labels on this github issue will be updated when the story is started.

[Johanna Ratliff]

@ashwithahn Is anything streaming? Are you able to cf logs bigapp2?

[ashwithahn]

@JohannaSmith : No. When i do cf logs bigapp2, i get connection timed out and no output. However, when cf logs bigapp2 --recent is done, a few lines of output is displayed

[Brady Love]

2018/07/17 04:08:37 websocket handler: Not a websocket handshake: websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header

Makes me thing that the WebSocket headers are not making to the TrafficController. @ashwithahn would you happen to be behind a proxy that could be removing the WebSocket client request headers?

What IaaS and LoadBalancer is your deployment using?

[ashwithahn]

Hi, I would like to close this issue. We resolved it by deploying aws.yml file separately. As cf-deploy was unable to make changes on port 4443 automatically.

@bradylove @JohannaSmith Thank you for replying :)