FIN packet is not appear on initaiation of GOAWAY in https case
MohammadNC opened this issue · comments
Need to close the netty connection gracefully on invocation of GOAWAY request
My application server using http2 reactive netty server version details are mentioned below
+--- org.springframework.boot:spring-boot-starter-webflux:2.7.13
| +--- org.springframework.boot:spring-boot-starter:2.7.13 ()
| +--- org.springframework.boot:spring-boot-starter-json:2.7.13 ()
| +--- org.springframework.boot:spring-boot-starter-reactor-netty:2.7.13
| | --- io.projectreactor.netty:reactor-netty-http:1.0.33
| | +--- io.netty:netty-codec-http:4.1.93.Final -> 4.1.94.Final
Expected Behavior
On initiation of GOAWAY request, the connection should get closed and FIN packet should appear.
Actual Behavior
in log found the OUTBOUND GO_AWAY but immediately we found "Sending GOAWAY failed: lastStreamId"
Steps to Reproduce
ChannelHandlerContext ctx;
Http2FrameCodec codec = (Http2FrameCodec) ctx.channel().parent().pipeline().get(NettyPipeline.HttpCodec);
codec.write(ctx,
new DefaultHttp2GoAwayFrame(Http2Error.INTERNAL_ERROR),
ctx.newPromise());
ctx.flush();
- JVM version (
java -version): 17.0.1 2021-10-19 LTS - OS and version (eg.
uname -a): Linux
@MohammadNC Please provide reproducible example with Reactor Netty API.
The provided snippet uses Netty API for sending frames but not Reactor Netty API.
Hi @violetagg
please find the below Reactor netty code snippet
ChannelHandlerContext ctx
Connection connection = Connection.from(ctx.channel());
Mono<Void> writeGoAwayMono = Mono.defer(() -> {
if (!connection.isDisposed()) {
ByteBuf goAwayByteBuf = Unpooled.buffer();
connection.channel().writeAndFlush(new DefaultHttp2GoAwayFrame(Http2Error.INTERNAL_ERROR),
connection.channel().voidPromise());
return Mono.empty();
} else {
return Mono.empty();
}
});
writeGoAwayMono.subscribe();
connection.channel().flush();
@MohammadNC You cannot use connection.channel().writeAndFlush(). As I said you need to use Reactor Netty API. If you use Netty API you bypass everything in Reactor Netty. Try to use connection.dispose()
Also explain in more details (or complete example) when and why do you want to close the connection.
HI @violetagg
as suggested by you updated code snippet
Connection connection = Connection.from(ctx.channel());
Mono<Void> writeGoAwayMono = Mono.defer(() -> {
if (!connection.isDisposed()) {
connection.dispose();
return Mono.empty();
} else {
return Mono.empty();
}
});
writeGoAwayMono.subscribe();
connection.channel().flush();
with these changes, I didn't find any logs related to GOAWAY and there is no FIN frame in the TCP dump.
The requirement is to close the netty connection and create a new one whenever https certificates are updated.
for this, the following code changes are done, please check and let me know how we can resolve this issue.
@Component("resetConnectionService")
public class ResetConnectionService extends ChannelInboundHandlerAdapter {
private boolean clientAuth = true;
Set<ChannelHandlerContext> channelHandlers = ConcurrentHashMap.newKeySet();
@Override
public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
channelHandlers.add(ctx);
}
@Override
public void handlerRemoved(ChannelHandlerContext ctx) throws Exception {
channelHandlers.remove(ctx);
}
@Override
public boolean isSharable() {
return true;
}
public void reloadCertificate(String certificateType) {
Ssl ssl = new Ssl();
ssl.setEnabled(true);
ssl.setKeyStore(keyStorePrimaryPath);
// enable/disable Mutual TLS
ssl.setClientAuth((clientAuth) ? ClientAuth.NEED : ClientAuth.NONE);
ssl.setKeyStorePassword(..);
ssl.setTrustStore(..);
.
ssl.setTrustStorePassword(..);
Http2 http2 = new Http2();
http2.setEnabled(true);
ExtendedSslServerCustomizer sslCustomizer = new ExtendedSslServerCustomizer(ssl, http2, null);
SslContextBuilder contextBuilder = sslCustomizer.getSslContextBuilder();
reloadDefaultConfiguration(contextBuilder);
SslProviderObject.getSslProvider().setSslContext(contextBuilder.build());
closeAllClientConnections();
}
private void closeAllClientConnections() {
if (channelHandlers.isEmpty()) {
return;
}
Set<ChannelHandlerContext> ctxList = new LinkedHashSet<ChannelHandlerContext>(channelHandlers);
for (ChannelHandlerContext ctx : ctxList) {
Connection connection = Connection.from(ctx.channel());
Mono<Void> writeGoAwayMono = Mono.defer(() -> {
if (!connection.isDisposed()) {
connection.dispose();
return Mono.empty();
} else {
return Mono.empty();
}
});
writeGoAwayMono.subscribe();
connection.channel().flush();
}
}
private void reloadDefaultConfiguration(SslContextBuilder sslContextBuilder) {
sslContextBuilder.sslProvider(
OpenSsl.isAvailable() ?
io.netty.handler.ssl.SslProvider.OPENSSL :
io.netty.handler.ssl.SslProvider.JDK);
}
}
The requirement is to close the netty connection and create a new one whenever https certificates are updated.
for this, the following code changes are done, please check and let me know how we can resolve this issue.
Just one clarification. When the old connections were opened they were opened with a valid certificate, but your requirement is not only to use the new certificate for the new connections but also to close the already opened connections regardless that the certificate was valid when they were opened?
Do you want a graceful shutdown for the old connections as most probably they are processing requests? Are we speaking for server or client side?
Hi @violetagg
yes, we need to gracefully close the old connection and I am using Netty for the server side
@MohammadNC May be you can try the suggestions below (this is just an idea, as I don't have any reproducible example I cannot guarantee that this will be a solution for your use case):
-
Enable graceful shutdown functionality provided by Spring Boot. This will guarantee that the server will notify the client to stop opening streams for a concrete connection and all current streams will be processed before closing the connection.
-
Instead of
for (ChannelHandlerContext ctx : ctxList) {
Connection connection = Connection.from(ctx.channel());
Mono<Void> writeGoAwayMono = Mono.defer(() -> {
if (!connection.isDisposed()) {
connection.dispose();
return Mono.empty();
} else {
return Mono.empty();
}
});
writeGoAwayMono.subscribe();
connection.channel().flush();
}
Try calling directly close. close knows that it must wait for the current streams to be processed.
for (ChannelHandlerContext ctx : ctxList) {
cix.close()
}
- You may need to configure also
closeNotifyReadTimeout. See more information here https://projectreactor.io/docs/netty/release/reference/index.html#http-server-ssl-tls-timeout
@MohammadNC Were you able to test the suggestions?
Hi @violetagg ,
I tried the recommended suggestion, but still no luck. here future.isSuccess() is true but there is no FIN in TCP and there is no log details related to GOAWAY.
for (ChannelHandlerContext ctx : ctxList) {
ctx.close().addListener(future -> {
if (future.isSuccess()) {
logger.warn("Channel closed successfully");
} else {
logger.warn("Channel close failed: {}", future.cause());
}
});
}
@MohammadNC Please enable wiretap and provide the logs or a reproducible example.
Hi @violetagg
Please find the attached logs, and pls provide your suggestions to close the netty connections.
Log_to_upload.zip
HI @violetagg
please let me know if any suggestions
HI @violetagg
please let me know if any suggestions.
@MohammadNC With your example, I'm seeing:
2023-10-17 11:01:30.122 INFO 83513 --- [ionShutdownHook] o.s.b.w.embedded.netty.GracefulShutdown : Commencing graceful shutdown. Waiting for active requests to complete
2023-10-17 11:01:30.123 INFO 83513 --- [ netty-shutdown] o.s.b.w.embedded.netty.GracefulShutdown : Graceful shutdown complete
2023-10-17 11:01:30.123 DEBUG 83513 --- [myapp-h2c-nio-2] reactor.netty.http.server.h2 : [id: 0x05e1bbcf, L:/127.0.0.1:8080 - R:/127.0.0.1:64717] OUTBOUND GO_AWAY: lastStreamId=5 errorCode=0 length=0 bytes=
One thing that I changed in your example is how you add your handler. Currently you are adding the handler on the level of stream, while you need to add it on the level of connection.
I did the following change:
factory.addServerCustomizers(httpServer ->httpServer.doOnChannelInit((obs, ch, addr) -> {
//conn.addHandlerLast(NettyCustomChannelHandler.INSTANCE);
Connection.from(ch).addHandlerLast(resetConnectionService);
}));
Hi @violetagg
Thank you for your suggestion. with your suggestions now FIN is appeared in TCP dump, but while executing the same suggestion in HTTPS environment getting following log.
{"instant":{"epochSecond":1697546132,"nanoOfSecond":406122663},"thread":"ingress-h2-epoll-1","level":"DEBUG","logg erName":"reactor.netty.http.server.h2","message":"[id: 0xc546f5e6, L:/192.168.219.111:9443 ! R:/192.168.219.69:530 96] OUTBOUND GO_AWAY: lastStreamId=2147483647 errorCode=2 length=24 bytes=53534c456e67696e6520636c6f73656420616c72 65616479","endOfBatch":false,"loggerFqcn":"io.netty.util.internal.logging.LocationAwareSlf4JLogger","threadId":104 ,"threadPriority":5}
{"instant":{"epochSecond":1697546132,"nanoOfSecond":406236108},"thread":"ingress-h2-epoll-1","level":"DEBUG","logg erName":"reactor.netty.http.server.HttpServer","message":"[c546f5e6, L:/192.168.219.111:9443 ! R:/192.168.219.69:5 3096] WRITE: 17B\n +-------------------------------------------------+\n | 0 1 2 3 4 5 6 7 8 9 a b c d e f |\n+--------+-------------------------------------------------+----------------+\n|000000 00| 00 00 20 07 00 00 00 00 00 7f ff ff ff 00 00 00 |.. .............|\n|00000010| 02 |. |\n+--------+-------------------------------------------------+----------------+ ","endOfBatch":false,"loggerFqcn":"io.netty.util.internal.logging.LocationAwareSlf4JLogger","threadId":104,"thread Priority":5}
{"instant":{"epochSecond":1697546132,"nanoOfSecond":406328740},"thread":"ingress-h2-epoll-1","level":"DEBUG","logg erName":"reactor.netty.http.server.HttpServer","message":"[c546f5e6, L:/192.168.219.111:9443 ! R:/192.168.219.69:5 3096] WRITE: 24B\n +-------------------------------------------------+\n | 0 1 2 3 4 5 6 7 8 9 a b c d e f |\n+--------+-------------------------------------------------+----------------+\n|000000 00| 53 53 4c 45 6e 67 69 6e 65 20 63 6c 6f 73 65 64 |SSLEngine closed|\n|00000010| 20 61 6c 72 65 61 64 79 | already |\n+--------+-------------------------------------------------+----------------+ ","endOfBatch":false,"loggerFqcn":"io.netty.util.internal.logging.LocationAwareSlf4JLogger","threadId":104,"thread Priority":5}
{"instant":{"epochSecond":1697546132,"nanoOfSecond":415830381},"thread":"ingress-h2-epoll-1","level":"DEBUG","logg erName":"reactor.netty.channel.ChannelOperations","message":"[c546f5e6/1077-1, L:/192.168.219.111:9443 ! R:/192.16 8.219.69:53096] An outbound error could not be processed","thrown":{"message":"SSLEngine closed already","name":"i o.netty.handler.ssl.SslClosedEngineException","extendedStackTrace":[{"class":"io.netty.handler.ssl.SslHandler","me thod":"wrap","file":"SslHandler.java","line":861}]},"endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4 jLogger","threadId":104,"threadPriority":5}
here in logs I am getting errorCode=2.
can you please check this once and let me know what might be the issue for this.
@MohammadNC With the provided example if I enable H2 instead of H2C, I cannot reproduce. Please advise how to change the example or provide a new example. Also the current example uses an old Spring Boot version 2.7.9, please ensure that you are using the latest.
Hi @violetagg ,
please find the attached sample test https program, for the further investigation.
httpsnettytest.zip
please let me know if any suggestions.
Hi @violetagg
please let me know if any suggestions.
@MohammadNC This example is strange. I have two servers started - one on port 8080 and one on port 9443 and I don't see how the graceful shutdown is enabled for the server that is listening on 9443.
Also I see some classes from Netty and Reactor Netty packaged within the example.
I apologise for the delayed reply.
@MohammadNC I think I found the issue. Ignore my comment above.
In the example, where you configure reloadServerCertificate, when it is H2 you need to configure it with the code below:
factory.addServerCustomizers(httpServer ->httpServer.doOnChannelInit((obs, ch, addr) -> {
Connection.from(ch).addHandlerLast(new ChannelInboundHandlerAdapter() {
@Override
public void userEventTriggered(ChannelHandlerContext ctx, Object evt) {
if (evt instanceof SslHandshakeCompletionEvent) {
ctx.channel().pipeline().remove(this);
Connection.from(ch).addHandlerLast(reloadServerCertificate);
}
ctx.fireUserEventTriggered(evt);
}
});
}));
Please give it a try and tell me the result.
Hi @violetagg
with the above suggestions connections are going to TIME_WAIT status after reloading the certificates. and in the subsequent request also connections are not establishing they are aging going to TIME_WAIT status.
Before reload
After reload of certitficates.
After reload of certificates, started the fresh traffic for this also connections are going to TIME_WAIT instead of ESTABLISH
please check this.
Note: with the above changes earlier errorCode=2 issue resolved but connections are going to TIME_WAIT
@violetagg
please let me know if any update.
@MohammadNC With the provided example I cannot reproduce any problem. I do see that the connection stays for a while in state TIME_WAIT but that's normal as there is a timeout
see https://en.wikipedia.org/wiki/Transmission_Control_Protocol
TIME-WAIT Server or client Waiting for enough time to pass to be sure that all remaining packets on the connection have expired.
@MohammadNC I'm closing this issue as I do not see any problem with Reactor Netty.