uber / tchannel

network multiplexing and framing protocol for RPC

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Hyperbahn: clean deploys

Raynos opened this issue · comments

The following things are bad during deploys.

Fix fatal exception

TchannelResponseAlreadyDoneError: cannot send send error frame: UnexpectedError: response already started (state 3), response already done in state: 3
    at Object.createError [as ResponseAlreadyDone] (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/node_modules/error/typed.js:31:22)
    at V2OutResponse.sendError (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/out_response.js:193:43)
    at TChannelV2Handler.onReqError (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/v2/handler.js:896:17)
    at DefinedEvent.onReqError [as listener] (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/v2/handler.js:879:14)
    at DefinedEvent.emit (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/lib/event_emitter.js:86:14)
    at TChannelConnection.buildResponse (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/connection_base.js:130:24)
    at buildResponse (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/connection_base.js:121:21)
    at onArg23 (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/request-handler.js:57:19)
    at compatCall (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/streaming_in_request.js:106:9)
    at zalgoSafe (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/node_modules/run-parallel/node_modules/dezalgo/dezalgo.js:20:10)
    at done (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/node_modules/run-parallel/index.js:18:15)
    at signalReady (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/ready-signal/index.js:21:26)
    at StreamArg.finish (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/argstream.js:257:9)
    at StreamArg.emit (events.js:117:20)
    at _stream_readable.js:943:16
    at process._tickDomainCallback (node.js:463:13)

Too high logger warn() during deploys

We get a lot of destroying due to init timeout logger warnings.

I suspect that when we close a connection or a connection closes we do not clear the init timeout marker in the timeout heap.

This is definitely a bug.