Cores on nginx-quiche while running curl command
Karthikdasari0423 opened this issue · comments
Hi ,
I tried to build nginx with quiche as explained below and tried to run curl with http3 but noticing below cores
[https://github.com/cloudflare/quiche/tree/master/nginx]
nginx -V
root@ubuntu:/tmp/cores# /src/nginx-1.16.1/objs/nginx -V
nginx version: nginx/1.16.1 (quiche-ab2b44fa)
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --prefix=/src/nginx-1.16.1 --build=quiche-ab2b44fa --with-http_ssl_module --with-http_v2_module --with-http_v3_module --with-openssl=../quiche/quiche/deps/boringssl --with-quiche=../quiche
curl command and version
root@ubuntu:/tmp/cores# curl -V
curl 8.3.0-DEV (x86_64-pc-linux-gnu) libcurl/8.3.0-DEV OpenSSL/1.1.1s zlib/1.2.11 brotli/1.0.9 msh3/0.6.0.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe UnixSockets
root@ubuntu:/tmp/cores# curl -k -v --http3-only -# -o /tmp/index.html https://127.0.0.1:5443/index.html
below is the nginx bt
(gdb) bt
#0 0x000055d6712442c2 in ngx_quic_write_handler (wev=<optimized out>) at src/event/ngx_event_quic.c:411
#1 0x000055d67123456f in ngx_event_process_posted (cycle=cycle@entry=0x55d6737d0700, posted=0x55d6717a0280 <ngx_posted_events>)
at src/event/ngx_event_posted.c:34
#2 0x000055d67123405c in ngx_process_events_and_timers (cycle=cycle@entry=0x55d6737d0700) at src/event/ngx_event.c:267
#3 0x000055d67123bf51 in ngx_worker_process_cycle (cycle=0x55d6737d0700, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:750
#4 0x000055d67123a431 in ngx_spawn_process (cycle=cycle@entry=0x55d6737d0700,
proc=proc@entry=0x55d67123be67 <ngx_worker_process_cycle>, data=data@entry=0x4, name=name@entry=0x55d6715d8236 "worker process",
respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:199
#5 0x000055d67123b537 in ngx_start_worker_processes (cycle=cycle@entry=0x55d6737d0700, n=8, type=type@entry=-3)
at src/os/unix/ngx_process_cycle.c:359
#6 0x000055d67123c62d in ngx_master_process_cycle (cycle=cycle@entry=0x55d6737d0700) at src/os/unix/ngx_process_cycle.c:131
#7 0x000055d671214e0f in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:382
(gdb) quit
nginx error log file
2023/09/06 11:26:06 [alert] 240245#0: worker process 240272 exited on signal 11 (core dumped)
2023/09/06 11:26:08 [alert] 240245#0: worker process 240274 exited on signal 11 (core dumped)
2023/09/06 11:26:12 [alert] 240245#0: worker process 240276 exited on signal 11 (core dumped)
2023/09/06 11:26:26 [alert] 240245#0: worker process 240247 exited on signal 11 (core dumped)
2023/09/06 11:26:27 [alert] 240245#0: worker process 240303 exited on signal 11 (core dumped)
2023/09/06 11:26:29 [alert] 240245#0: worker process 240305 exited on signal 11 (core dumped)
2023/09/06 11:26:33 [alert] 240245#0: worker process 240307 exited on signal 11 (core dumped)
2023/09/06 11:31:05 [alert] 240245#0: worker process 240250 exited on signal 11 (core dumped)
2023/09/06 11:31:06 [alert] 240245#0: worker process 241691 exited on signal 11 (core dumped)
2023/09/06 11:31:08 [alert] 240245#0: worker process 241692 exited on signal 11 (core dumped)
2023/09/06 11:31:12 [alert] 240245#0: worker process 241693 exited on signal 11 (core dumped)
2023/09/06 11:38:52 [alert] 240245#0: worker process 241694 exited on signal 11 (core dumped)
2023/09/06 11:38:53 [alert] 240245#0: worker process 241734 exited on signal 11 (core dumped)
2023/09/06 11:38:55 [alert] 240245#0: worker process 241735 exited on signal 11 (core dumped)
2023/09/06 11:38:59 [alert] 240245#0: worker process 241736 exited on signal 11 (core dumped)
root@ubuntu:/tmp/cores#
am i missing anything here?
Hello, I encountered some problems when configuring the quic service of NGINX. My configuration is the same as that of the official website, but I still cannot use the quic protocol when accessing the server. The h2 protocol is still used. Here is my compilation information:
nginx version: nginx/1.16.1 (quiche-83d9168a)
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --build=quiche-83d9168a --with-http_v3_module --with-http_v2_module --with-quiche=/quiche --with-http_ssl_module --with-openssl=/quiche/quiche/deps/boringssl --add-module=/connect
i build nginx with below args
root@ubuntu:~# /src/nginx-quiche/objs/nginx -V
nginx version: nginx/1.16.1 (quiche-cf2a087)
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --prefix=/src/nginx-quiche --build=quiche-cf2a087 --with-http_ssl_module --with-http_v2_module --with-http_v3_module --with-openssl=../quiche/deps/boringssl --with-quiche=../quiche
root@ubuntu:~#
Can you share the nginx conf file and command you are using to connect to nginx?
This is my configuration information：
server {
# Enable QUIC and HTTP/3.
listen 443 quic reuseport;
server_name test.cn;
# Enable HTTP/2 (optional).
listen 443 ssl http2;
ssl_certificate /usr/local/nginx/conf/cert/test.pem;
ssl_certificate_key /usr/local/nginx/conf/cert/test.key;
# Enable all TLS versions (TLSv1.3 is required for QUIC).
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
# Add Alt-Svc header to negotiate HTTP/3.
add_header alt-svc 'h3=":443"; ma=86400';
}
Can you help me see what the problem is?
can you add below header also
add_header X-protocol $server_protocol always;
and can you send netstat -alpn | grep nginx output
Here are the details：
root@learn:/quiche# netstat -alpn |grep nginx
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 20157/nginx: master
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 20157/nginx: master
udp 0 0 0.0.0.0:443 0.0.0.0:* 20157/nginx: master
unix 3 [ ] 流 已连接 928126 20157/nginx: master
unix 3 [ ] 流 已连接 928127 20157/nginx: master
can you try below config
it works for me
# to use the same port for quic and https
listen 5443 quic reuseport;
listen [::]:5443 quic reuseport;
listen 5443 ssl;
ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
ssl_protocols TLSv1.3;
ssl_ciphers ALL:COMPLEMENTOFALL;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets on;
ssl_session_timeout 5m;
#to enable 0-rtt
ssl_early_data off;
#to change to /var/www/html/
root /var/www/html/;
location / {
# required for browsers to direct them into quic port
add_header Alt-Svc 'h3=":$server_port"; ma=86400';
#add_header Alt-Svc 'h3=":5443"; ma=86400';
# signal whether we are using QUIC+HTTP/3
add_header X-protocol $server_protocol always;
}
After using this configuration, he compiled without problems, but the access times error "404".Is there something I'm missing?
can you send me client output
Here are details:
"192.168.227.1 - - [04/Dec/2023:20:12:08 +0800] "GET / HTTP/1.1" 404 555 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
192.168.227.1 - - [04/Dec/2023:20:14:18 +0800] "GET / HTTP/2.0" 404 555 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36""
seems to me this is nginx error log file output
I don't quite understand what you mean. Here is the message I get back when I pass the "crul" command, hopefully I understood it correctly：
C:\Users\dell>curl -v https://ryetong.cn
- Trying 192.168.227.129:443...
- Connected to ryetong.cn (192.168.227.129) port 443 (#0)
- schannel: disabled automatic use of client certificate
- ALPN: offers http/1.1
- ALPN: server accepted http/1.1
- using HTTP/1.1
GET / HTTP/1.1
Host: ryetong.cn
User-Agent: curl/8.0.1
Accept: /
- schannel: remote party requests renegotiation
- schannel: renegotiating SSL/TLS connection
- schannel: SSL/TLS connection renegotiated
< HTTP/1.1 404 Not Found
< Server: nginx/1.16.1
< Date: Mon, 04 Dec 2023 12:48:01 GMT
< Content-Type: text/html
< Content-Length: 153
< Connection: keep-alive
< X-protocol: HTTP/1.1
<
404 Not Found
nginx/1.16.1 * Connection #0 to host ryetong.cn left intact
C:\Users\dell>curl -v https://test.cn
- Trying 192.168.227.163:443...
- Connected to test.cn (192.168.227.163) port 443 (#0)
- schannel: disabled automatic use of client certificate
- ALPN: offers http/1.1
- ALPN: server accepted http/1.1
- using HTTP/1.1
GET / HTTP/1.1
Host: ryetong.cn
User-Agent: curl/8.0.1
Accept: /
- schannel: remote party requests renegotiation
- schannel: renegotiating SSL/TLS connection
- schannel: SSL/TLS connection renegotiated
< HTTP/1.1 404 Not Found
< Server: nginx/1.16.1
< Date: Mon, 04 Dec 2023 12:50:02 GMT
< Content-Type: text/html
< Content-Length: 153
< Connection: keep-alive
< X-protocol: HTTP/1.1
<
404 Not Found
nginx/1.16.1 * Connection #0 to host ryetong.cn left intact
i think you didn't even tried with http3
try as curl --http3-only -v -k https://ryetong.cn:5443/
and does your curl support quic?
curl -V output ?
Sorry, my "curl" does not support http3：
curl: option --http3-only: the installed libcurl version doesn't support this
curl: try 'curl --help' for more information
try to install curl with h3 support
@Karthikdasari0423 Have you solved this coredump problem? How to solve it?
@pplabs-fute yes,just install with https://github.com/bagder/quiche.git instaed of original one
@Karthikdasari0423 ok，thanks;
@pplabs-fute yes,just install with https://github.com/bagder/quiche.git instaed of original one
Hey, I've had this issue too for a long time and it seems that using this fork to build nginx fixes the issue - so thanks haha.
Do you have any idea why this fork works but the official repo doesn't? And how did you find out this fork works?
@bagder Seems to me you have deleted https://github.com/bagder/quiche.git
Could you pls add back if you archived or made it private?
Many people are using your repo as it is stable in building nginx with quiche support
Sorry, my fork is gone.
okay, got it.
Thanks @bagder
I've been looking into this issue and was able to solve it by changing this piece of code:
Lines 1050 to 1054 in 1780aec
The segfault comes from the quic_blocked_events queue which is uninitialised. But the fix is not to initialise it per se.
The real problem seems to me that consulting the write event of the listening connection is not appropriate to determine whether the connection socket is ready to write.
Changing the if condition to if (!c->write->ready) makes the code run and the QUIC connection go on.
@bagder could you kindly confirm this is the fix you undertook? Thanks :)
@bagder could you kindly confirm this is the fix you undertook? Thanks :)
I did not (on purpose) fix this. I suspect maybe my fork was just out of date.
@mpiraux Does your below fork works for building nginx with quiche support?
https://github.com/mpiraux/quiche
and i have a backup of @bagder fork and as @bagder said it is out of date.
root@ubuntu:~/quiche# git log
commit cf2a08757c942d13f15a5a22aa7ea9ef50309cbe (HEAD -> master, origin/master, origin/HEAD)
Author: Junho Choi <junho@cloudflare.com>
Date: Thu May 27 15:56:09 2021 -0700
cubic: fix cwnd growth during congestion avoidance
cwnd_inc is used for storing cwnd increments during congestion
avoidance. When cwnd_inc >= MSS we increase cwnd by 1 MSS.
Currently we clear cwnd_inc when cwnd is updated but this will
lead to slightly slower growth because the residual part is gone.
commit 059b3d9c333ba61b2dc01f5e14ef95badca4fa03
Author: Junho Choi <1229714+junhochoi@users.noreply.github.com>
Date: Fri May 28 01:50:36 2021 -0700
don't try to send HANDSHAKE_DONE on the client
`HANDSHAKE_DONE` can only be sent by a server, so don't try to send a packet from the client if the frame hasn't been sent yet.
I've been looking into this issue and was able to solve it by changing this piece of code:
Lines 1050 to 1054 in 1780aec
The segfault comes from the
quic_blocked_eventsqueue which is uninitialised. But the fix is not to initialise it per se.The real problem seems to me that consulting the write event of the listening connection is not appropriate to determine whether the connection socket is ready to write.
Changing the if condition to
if (!c->write->ready)makes the code run and the QUIC connection go on.@bagder could you kindly confirm this is the fix you undertook? Thanks :)
and this code is not even present in @bagder fork