Pinging idle connection before sending request is ignored and node process exits unexpectedly
jeanp413 opened this issue · comments
Describe the bug
After being idle from a long time and calling a method, the second call is skipped completely and the node process exits unexpectedly.
This is because the ping call in the verify state is done with the socket unrefed.
To Reproduce
Tested this on linux using latest version:
const transport = createConnectTransport({
baseUrl: serviceUrl.toString(),
httpVersion: '2',
interceptors: [authInterceptor],
useBinaryFormat: true,
pingIntervalMs: 120000,
});
const service = createPromiseClient(MyService, transport);
const response1 = await service.method({});
console.log(`===> ${new Date()} On method response}`, response1);
console.log(`===> ${new Date()} Going to sleep`);
await timeout(10*60*1000);
console.log(`===> ${new Date()} Awake`);
// This await is ignored and the node process exits unexpectedly
const response2 = await service.method({});
console.log(`===> ${new Date()} On method response2}`, response2);Additional context
Add any other context about the problem here.
Hey! Can you give the node version that you are using, I am unable to reproduce using node 20.7.0 with:
const transport = createConnectTransport({
baseUrl: "https://demo.connectrpc.com",
httpVersion: "2",
useBinaryFormat: true,
pingIntervalMs: 120000,
});
const service = createPromiseClient(ElizaService, transport);
const response1 = await service.say({ sentence: "Hello" });
console.log(`===> ${new Date()} On method response}`, response1);
console.log(`===> ${new Date()} Going to sleep`);
await new Promise((resolve) => setTimeout(resolve, 10 * 60 * 1000));
console.log(`===> ${new Date()} Awake`);
const response2 = await service.say({});
console.log(`===> ${new Date()} On method response2}`, response2);We don't send ping frames for idle connections, unless pingIdleConnection is set to true along with pingIntervalMs. So it should not be a ping issue in this case. Regarding the unref, it should not be a problem here, as the execution of the call itself will result in events. Besides, we call ref before we send any ping frames when we have an active stream:
Can you also try with the sample I pasted and let us know the results, it is similar I just removed the auth interceptor and used the demo server.
@srikrsna-buf I'm using node 18 and ubuntu
We don't send ping frames for idle connections, unless pingIdleConnection is set to true along with pingIntervalMs. So it should not be a ping issue in this case. Regarding the unref, it should not be a problem here, as the execution of the call itself will result in events. Besides, we call ref before we send any ping frames when we have an active stream:
When pingIntervalMs is specified a ping frame is send before doing the actual request to verify the connection, it goes to verifying state and the connection is unrefed in that case.
Can you also try with the sample I pasted and let us know the results, it is similar I just removed the auth interceptor and used the demo server.
Will try your sample 👍
When
pingIntervalMsis specified a ping frame is send before doing the actual request to verify the connection, it goes to verifying state and the connection is unrefed in that case.
Yes, but we wait for the verified promise to complete that should hold the event loop, but I see what your saying I'll try to trigger the verify in my test.
@srikrsna-buf I can repro with your sample, are you using Linux?
Me too I just got it to trigger the verify flow. I'll send a fix soon Thank you!
Thanks for the detailed report, @jeanp413 ❤️
@srikrsna-buf @timostamm looks like this wasn't the only bug, now the event loop doesn't exit but when going from ready state to verifying the this.s.onExitState?.() is invoked clearing the pingTimeoutId so the ping won't timeout after pingTimeoutMs (15 seconds by default), it will eventually timeout after around 17 minutes with Error [ERR_HTTP2_PING_CANCEL]: HTTP2 ping cancelled