Hyperbahn: clean deploys
Raynos opened this issue · comments
The following things are bad during deploys.
Fix fatal exception
TchannelResponseAlreadyDoneError: cannot send send error frame: UnexpectedError: response already started (state 3), response already done in state: 3
at Object.createError [as ResponseAlreadyDone] (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/node_modules/error/typed.js:31:22)
at V2OutResponse.sendError (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/out_response.js:193:43)
at TChannelV2Handler.onReqError (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/v2/handler.js:896:17)
at DefinedEvent.onReqError [as listener] (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/v2/handler.js:879:14)
at DefinedEvent.emit (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/lib/event_emitter.js:86:14)
at TChannelConnection.buildResponse (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/connection_base.js:130:24)
at buildResponse (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/connection_base.js:121:21)
at onArg23 (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/request-handler.js:57:19)
at compatCall (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/streaming_in_request.js:106:9)
at zalgoSafe (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/node_modules/run-parallel/node_modules/dezalgo/dezalgo.js:20:10)
at done (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/node_modules/run-parallel/index.js:18:15)
at signalReady (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/ready-signal/index.js:21:26)
at StreamArg.finish (/var/cache/udeploy/r/hyperbahn/sjc1-produ-0000000045-v1/node_modules/hyperbahn/node_modules/tchannel/argstream.js:257:9)
at StreamArg.emit (events.js:117:20)
at _stream_readable.js:943:16
at process._tickDomainCallback (node.js:463:13)
Too high logger warn() during deploys
We get a lot of destroying due to init timeout
logger warnings.
I suspect that when we close a connection or a connection closes we do not clear the init timeout marker in the timeout heap.
This is definitely a bug.