reactor netty (1.0.33) not able to decode the packet with PRI* method [PRI* HTTP/2.0] in HTTPS connection
MaheshJob opened this issue · comments
While client is sending a HTTPS request with OPTIONS method and HTTP/2.0 to server (reactor-netty), the TLS handshake is successful but the request message is not able reach the application. server is using reactor netty (reactor-netty-http:1.0.33) and I can see that reactor netty is not able to decode the packet which is coming with "PRI" method as shown in the below logs.
{"epochSecond":1699942246,"nanoOfSecond":770639339}
,"thread":"ingress-h2-epoll-4","level":"DEBUG","loggerName":"reactor.netty.http.server.HttpServer","message":"[5130ae29, L:/10.233.88.131:9443 - R:/10.233.96.60:35244 ] USER_EVENT: SslHandshakeCompletionEvent(SUCCESS)","endOfBatch":false,"loggerFqcn":"io.netty.util.internal.loggin g.LocationAwareSlf4JLogger","threadId":375,"threadPriority":5,"messageId":"","messageTimestamp":"2023-11-14T06:10:46.770+0000","application":"testApp","m icroservice":"test-work","engVersion":"1.2.3","mktgVersion":"1.2","namespace":"test","node":"test","pod":"test","subsystem":"test","instanceType":"prod","processId":"1"}
{"instant":
{"epochSecond":1699942246,"nanoOfSecond":772461268}
,"thread":"ingress-h2-epoll-4","level":"WARN","loggerName":"reactor.netty.http.server.HttpServerOperations","message":"[5130ae29, L:/10.233.88.131:9443 - R:/10.233.96.60:35244] Decoding failed: REQUEST(decodeResult: failure(java.lang.IllegalStateException: Unexpected request [PRI* HTTP/2.0]), version: HTTP/2.0)\nPRI * HTTP/2.0","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4j Logger","threadId":375,"threadPriority":5,"messageId":"","messageTimestamp":"2023-11-14T06:10:46.772+0000","application":"testApp","microservice":"test-work","engVersion":"1.2.3","mktgVersion":"1.2","vendor":"test","namespace":"test","node":"test","pod":"test","subsystem":"test","instanceType":"prod","processId":"1"}
{"instant":
{"epochSecond":1699942246,"nanoOfSecond":773185503}
,"thread":"ingress-h2-epoll-4","level":"DEBUG","loggerName":"reactor.netty.http.server.HttpServer","message":"[5130ae29, L:/10.233.88.131:9443 ! R:/10.233.96.60:35244] USER_EVENT: SslCloseCompletionEvent(java.nio.channels.ClosedChannelException)","endOfBatch":false,"loggerFqcn":" io.netty.util.internal.logging.LocationAwareSlf4JLogger","threadId":375,"threadPriority":5,"messageId":"","messageTimestamp":"2023-11-14T06:10:46.773+0 000","application":"testApp","microservice":"test-work","engVersion":"1.2.3","mktgVersion":"1.2","vendor":" test","namespace":"test","node":"test","pod":"test","subsystem":"test","instanceType":"prod","processId":"1"}
could you pls check and let me know the reason for this decoding failure at reactor netty ? is there anything missing in the incoming request ?
Can you please share how it configured your reactor netty http server ?
especially:
- what protocol(s) is (are) passed to the HttpServer
protocolmethod ? - what ssl context is passed to the secure method HttpServer.secure(spec -> sslContext(...)) ?
thanks.
HTTP11, H2 protocols are passed to the HttpServer protocol method.
I am using reactor-netty-http:1.0.33 (with spring-webflux)..
+--- org.springframework.boot:spring-boot-starter-webflux:2.7.13
| +--- org.springframework.boot:spring-boot-starter:2.7.13 ()
| +--- org.springframework.boot:spring-boot-starter-json:2.7.13 ()
| +--- org.springframework.boot:spring-boot-starter-reactor-netty:2.7.13
| | --- io.projectreactor.netty:reactor-netty-http:1.0.33
I am using NettyReactiveWebServerFactory to set Http2 and SSL, pls find the below code snippet for the same.
try {
logger.debug("Enabling https port " + httpsPort + " on netty server");
LoopResources resource = LoopResources.create("ingress-h2");
ReactorResourceFactory reactorResourceFactory = new ReactorResourceFactory();
reactorResourceFactory.setLoopResources(resource);
reactorResourceFactory.setUseGlobalResources(false);
NettyReactiveWebServerFactory factory = new NettyReactiveWebServerFactory(httpsPort);
factory.setResourceFactory(reactorResourceFactory);
WorkerSSLConfig workerSLConfig = CommonUtils.getConfigMapFromYaml(
"application/application", WorkerSSLConfig.class);
WorkerSslStoreProp keyStoreProp = workerSLConfig.getKeyStoreServer();
WorkerSslStoreProp trustStoreProp = workerSLConfig.getTrustStoreServer();
Ssl ssl = new Ssl();
ssl.setEnabled(true);
// enable/disable Mutual TLS
ssl.setClientAuth((clientAuth) ? Ssl.ClientAuth.NEED : Ssl.ClientAuth.NONE);
ssl.setKeyStore(keyStoreProp.getPrimaryPath());
ssl.setKeyStorePassword(workerSLConfig.getStorePassword(keyStoreProp));
ssl.setTrustStore(trustStoreProp.getPrimaryPath());
ssl.setTrustStorePassword(workerSLConfig.getStorePassword(trustStoreProp));
if(keepAliveEnabled && Epoll.isAvailable()) {
factory.addServerCustomizers(builder -> builder.tcpConfiguration(tcpServer ->
tcpServer
.option(ChannelOption.SO_KEEPALIVE, true)
.option(EpollChannelOption.TCP_KEEPIDLE,keepAliveIdleTime)
.option(EpollChannelOption.TCP_KEEPCNT,keepAliveProbe)
.option(EpollChannelOption.TCP_KEEPINTVL,keepAliveInterval)));
}
if(workerSLConfig.getCiphers() != null && workerSLConfig.getCiphers().length>0) {
Http2SecurityUtil.CIPHERS.clear();
for (String cipher : workerSLConfig.getCiphers()) {
Http2SecurityUtil.CIPHERS.add(cipher);
}
}
Http2 http2 = new Http2();
http2.setEnabled(true);
factory.setPort(httpsPort);
factory.setSsl(ssl);
factory.setHttp2(http2);
factory.addServerCustomizers(server -> server.
tcpConfiguration(tcpServer-> tcpServer.doOnConnection(connection -> {
connection.addHandlerLast(reloadServerCertificate);
ChannelFuture closeFuture = connection.channel().closeFuture();
closeFuture.addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture future) throws Exception {
if(!future.isSuccess()) {
String host = ((InetSocketAddress)future.channel().remoteAddress()).getHostString();
String port = String.valueOf(((InetSocketAddress)future.channel().remoteAddress()).getPort());
}
}
});
})));
factory.addServerCustomizers(httpServer -> httpServer.wiretap(true));
factory.addServerCustomizers(httpServer -> httpServer.doOnConnection(conn -> conn.addHandlerLast(TestNettyCustomChannelHandler.INSTANCE)));
this.http = factory.getWebServer(this.httpHandler);
this.http.start();
} catch (Exception e) {
String logMsg = LogMessage.getErrorMsg(LogMsgCategory.INTERNAL,
WorkerLogMsgEvent.INITIALIZATION_FAILED, e,
"start(): Netty Server Initialization Exception");
logger.error(logMsg);
throw e;
}
I have tried HTTPS request with GET,PUT etc Methods and its working fine but HTTPS request with OPTIONS method is giving exception.
Could you please check and let me know the reason for failure and do the needful ?
am I missing something or isn't it that the H2 protocol is not declared using any addServerCustomizers using protocol method ?
can you add this customizer ? (or I'm missing it ?)
factory.addServerCustomizers(builder -> builder.protocol(HttpProtocol.HTTP11, HttpProtocol.H2));
The IllegalStateException: Unexpected request [PRI* HTTP/2.0]), version: HTTP/2.0)\nPRI *is most likely caused by the fact that the HttpProtocol.H2 is not enabled in the HttpServer using HttpServer.protocol method.
For example, the following sample code works for me:
@Configuration
public class NettyHttpServerConfig implements WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {
@Value("${server.http.port}")
private Integer httpPort;
@Override
public void customize(NettyReactiveWebServerFactory factory) {
LoopResources resource = LoopResources.create("myapp-h2c");
ReactorResourceFactory reactorResourceFactory = new ReactorResourceFactory();
reactorResourceFactory.setLoopResources(resource);
reactorResourceFactory.setUseGlobalResources(false);
factory.setResourceFactory(reactorResourceFactory);
// Set Port and enable HTTP/2
Http2 h2 = new Http2();
h2.setEnabled(true);
factory.setPort(httpPort);
factory.setHttp2(h2);
// Create EpollEventLoopGroup with size set to cores available in the system
if (Epoll.isAvailable()) {
EventLoopGroup boss = new EpollEventLoopGroup(Runtime.getRuntime().availableProcessors());
factory.addServerCustomizers(httpServer -> httpServer.runOn(boss));
System.out.println("Netty Epoll Running");
} else {
System.out.println("Netty NioEventLoop Running");
}
// HTTP 2.0 settings
SelfSignedCertificate cert = null;
try {
cert = new SelfSignedCertificate();
} catch (Exception e) {
e.printStackTrace();
}
// this one
factory.addServerCustomizers(builder -> builder.protocol(HttpProtocol.HTTP11, HttpProtocol.H2));
factory.addServerCustomizers(httpServer -> httpServer.http2Settings(s -> s.maxConcurrentStreams(100)));
factory.addServerCustomizers(httpServer -> httpServer.idleTimeout(Duration.ofMillis(10000)));
SelfSignedCertificate CERT = cert;
factory.addServerCustomizers(builder -> builder.secure(spec -> spec.sslContext(Http2SslContextSpec.forServer(CERT.certificate(), CERT.privateKey()))));
factory.addServerCustomizers(httpServer -> httpServer.doOnConnection(conn -> conn.addHandlerLast(NettyCustomChannelHandler.INSTANCE)));
factory.addServerCustomizers(httpServer -> httpServer.wiretap(true));
factory.setShutdown(Shutdown.GRACEFUL);
}
static final class NettyCustomChannelHandler extends ChannelOutboundHandlerAdapter {
static final NettyCustomChannelHandler INSTANCE = new NettyCustomChannelHandler();
@Override
public boolean isSharable() {
return true;
}
}
}
and if I comment the factory.addServerCustomizers(builder -> builder.protocol(HttpProtocol.HTTP11, HttpProtocol.H2));line, then I'm reproduce the IllegalStateException exception.
Let me know.
Thank you so much for your quick response.
I wanted to confirm that, with my code given above is perfectly working for all the PUT, GET, POST, PATCH, DELETE methods but when I use OPTIONS method Reactor netty is throwing an error response.
can you please confirm that whether you are seeing the IllegalStateException exception for only OPTIONS method or for all the above mentioned methods(PUT, GET, POST, PATCH, DELETE) also?
Note: In my code, when I print the supported protocols in HttpServer protocol API(FYI, please see the below code), I can see HTTP11, H2 protocols in the logs.
public final HttpServer protocol(HttpProtocol... supportedProtocols) {
for (HttpProtocol protocol: supportedProtocols) {
System.out.println("Testing ####################"+protocol.name());
}
Objects.requireNonNull(supportedProtocols, "supportedProtocols");
HttpServer dup = duplicate();
dup.configuration().protocols(supportedProtocols);
return dup;
}
please ignore my suggestion from #2988 (comment)
now, I have tried to reproduce this problem, but it works for me
can you please check this attached repro.tgz and try to modify it in order to fully reproduce the problem ?
To test:
- generate the gradle wrapper (only the first time you compile)
gradle wrapper
- build using java8:
./gradlew build
- start the server:
java -jar build/libs/repro-0.0.1-SNAPSHOT.jar
- and do a test using cur:
curl -k -v --http2 -X OPTIONS https://127.0.0.1:8080/hello
The class to adapt is NettyHttpServerConfig. it's close to your initial sample code, except that no cyphers are used, and there is no special netty handlers.
Please try to adapt the repro so we can reproduce and investigate the issue.
thank you.
Hi,
Thanks for your response.
I have used your code and tried to reproduce the issue in the following 2 ways.
- when I try to test with simple curl command with HTTP2 OPTIONS it is working fine
- when I try to send a request with HTTP2 OPTIONS from another client to HttpServer , i am seeing the decoding failed.
and I observed a difference in the logs as below.
-
when I try to test with simple curl command with HTTP2 OPTIONS it is working fine : as part of this testing I can see the "Negotiated application-level protocol [h2]" as show in the following log and the request processing was successful.
{"instant":{"epochSecond":1701857706,"nanoOfSecond":685988017},"thread":"test-epoll-3","level":"DEBUG","loggerName":"reactor.netty.http.server .HttpServerConfig","message":"[3939c489, L:/10.244.202.142:9443 - R:/10.244.202.129:53876] Negotiated application-level protocol [h2]","endOfBatch": false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":523,"threadPriority":5,"messageId":"","messageTimestamp":"2023-12-06T10:15:06.685+0000","application":"testAPP1","microservice":"test1","engVersion":"24.test","mktgVersion":"24.test.0","namespace":"test-master1","node":"100.77.10.78","pod":"test1-767c5c9596-prr5j","subsystem":"testss1","instanceType":"prod","processId":"1"}
-
when I try to send a request HTTP2 OPTIONS from another client to HttpServer : as part of this testing I can see the Negotiated application-level protocol [http/1.1] as shown in the following log and the request processing was not successful due to failure in decoding of the request.
{"instant":{"epochSecond":1701856056,"nanoOfSecond":518842710},"thread":"test-epoll-1","level":"DEBUG","loggerName":"reactor.netty.http.serve r.HttpServerConfig","message":"[c13173ab, L:/10.244.202.142:9443 - R:/10.244.202.129:63304] Negotiated application-level protocol [http/1.1]","endO fBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":138,"threadPriority":5,"messageId":"","messageTimestamp":"2023-12-06T09:47:36.518+0000","application":"testapp","microservice":"test","engVersion":"23.test","mktgVersion":"23.test.0","namespace":"test-master","node":"100.77.10.78","pod":"test-767c5c9596-prr5j","subsystem":"testss","instanceType":"prod","processId":"1"}
In the second test case, why HttpServer is negotiating the protocol as [http/1.1], can you please explain/let me know the reasons for this ?
what is the other client ?
It seems that the other client could be wrongly configured: it indicates to the server that it supports protocol http/1.1 over TLS (using ALPN), but it however uses protocol H2, not HTTP/1.1 secure.
To reproduce the problem, here is a reactor netty http client that is wrongly configured:
When the client connects to the server, the server sees that, according to ALPN, the client supports HTTP/1.1. However, the client starts communication by sending an HTTP/2 request (PRI), contradicting the indicated protocol support, so the server logs the IllegalStateException and responds with a 400 bad request in http/1.1:
@Test
void testBadClient_That_Negociates_HTTP11_But_uses_H2_protocol() throws Exception {
// Http2SslContextSpec should be used because we use use protocol=H2 (see below)
Http11SslContextSpec clientCtx =
Http11SslContextSpec.forClient()
.configure(builder -> builder.trustManager(InsecureTrustManagerFactory.INSTANCE));
ConnectionProvider provider = ConnectionProvider.builder("testMe")
.build();
HttpClient client =
createClient(provider, 8080)
.protocol(HttpProtocol.H2)
.secure(spec -> spec.sslContext(clientCtx));
int status = client
.options()
.uri("https://localhost:8080/hello")
.responseSingle((response, byteBufMono) -> Mono.just(response.status().code()))
.log()
.block();
provider.dispose();
assertThat(status == 200).isTrue();
}
2023-12-06 17:27:33.730 DEBUG 20103 --- [ myloops-nio-3] r.netty.http.server.HttpServerConfig : [4c643a2c, L:/127.0.0.1:8080 - R:/127.0.0.1:54614] Negotiated application-level protocol [http/1.1]
2023-12-06 17:35:50.397 DEBUG 20103 --- [ myloops-nio-4] reactor.netty.http.server.HttpServer : [5f4586ae, L:/127.0.0.1:8080 - R:/127.0.0.1:54721] READ: 39B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 50 52 49 20 2a 20 48 54 54 50 2f 32 2e 30 0d 0a |PRI * HTTP/2.0..|
|00000010| 0d 0a 53 4d 0d 0a 0d 0a 00 00 06 04 00 00 00 00 |..SM............|
|00000020| 00 00 06 00 00 20 00 |..... . |
+--------+-------------------------------------------------+----------------+
2023-12-06 17:35:50.397 WARN 20103 --- [ myloops-nio-4] r.n.http.server.HttpServerOperations : [5f4586ae, L:/127.0.0.1:8080 - R:/127.0.0.1:54721] Decoding failed: REQUEST(decodeResult: failure(java.lang.IllegalStateException: Unexpected request [PRI * HTTP/2.0]), version: HTTP/2.0)
PRI * HTTP/2.0
2023-12-06 17:35:50.397 DEBUG 20103 --- [ myloops-nio-4] r.n.http.server.HttpServerOperations : [5f4586ae-1, L:/127.0.0.1:8080 - R:/127.0.0.1:54721] Detected non persistent http connection, preparing to close
2023-12-06 17:35:50.397 DEBUG 20103 --- [ myloops-nio-4] reactor.netty.http.server.HttpServer : [5f4586ae-1, L:/127.0.0.1:8080 - R:/127.0.0.1:54721] WRITE: 66B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 54 54 50 2f 31 2e 31 20 34 30 30 20 42 61 64 |HTTP/1.1 400 Bad|
|00000010| 20 52 65 71 75 65 73 74 0d 0a 63 6f 6e 74 65 6e | Request..conten|
|00000020| 74 2d 6c 65 6e 67 74 68 3a 20 30 0d 0a 63 6f 6e |t-length: 0..con|
|00000030| 6e 65 63 74 69 6f 6e 3a 20 63 6c 6f 73 65 0d 0a |nection: close..|
|00000040| 0d 0a |.. |
+--------+-------------------------------------------------+----------------+
So, if the client wants to use protocol H2, it should configure the sslContext with an Http2SslContextSpec instead of a Http11SslContextSpec.
The following fixed client works fine:
@Test
void test_consistent_ALPN_with_H2() throws Exception {
Http2SslContextSpec clientCtx =
Http2SslContextSpec.forClient()
.configure(builder -> builder.trustManager(InsecureTrustManagerFactory.INSTANCE));
// Http11SslContextSpec clientCtx =
// Http11SslContextSpec.forClient()
// .configure(builder -> builder.trustManager(InsecureTrustManagerFactory.INSTANCE));
ConnectionProvider provider = ConnectionProvider.builder("testMe")
.build();
HttpClient client =
createClient(provider, 8080)
.protocol(HttpProtocol.H2)
.secure(spec -> spec.sslContext(clientCtx));
int status = client
.options()
.uri("https://localhost:8080/hello")
.responseSingle((response, byteBufMono) -> Mono.just(response.status().code()))
.log()
.block();
provider.dispose();
assertThat(status == 200).isTrue();
}
now, I don't know what is your other client, and I suggest to check how ALPN is configured in that client, and if it's consistent with HTTP/2 protocol.
Hi @MaheshJob ,
I'm closing this issue for the moment, however we can reopen it if you have some updates.
thanks.