nginx 1.25.3.1 http2 segfault with ja3 patch added
macskas opened this issue · comments
I cannot reproduce the error with curl, I can't see it in the logs(segfaults before the log) and there are like 4k rps on a single server, so debug logs are not really an option :( In the core I see the orignal request url with gdb. But thats about it. If I call the url directly there is no error.
Today I tested with and without ja3 patch. Same build process (official openresty builder). With ja3, there is a segfault in every 10 minutes, without it no segfault at all.
The core contains sensitive information so I cannot share it publicly,
built with OpenSSL 1.1.1w 11 Sep 2023
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.3 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.26 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.37 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.20 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.14 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib -Wl,-rpath,' --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module
--with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --add-dynamic-module=/tmp/modules/brotli --add-dynamic-module=/tmp/modules/geoip2 --add-module=/tmp/modules/nginx-ssl-fingerprint --with-stream --without-pcre2 --with-http_ssl_module
I know this is not much:
(gdb) info shared
From To Syms Read Shared Object Library
0x00007eff33311220 0x00007eff33312179 Yes /lib/x86_64-linux-gnu/libdl.so.2
0x00007eff332f3ae0 0x00007eff33303535 Yes /lib/x86_64-linux-gnu/libpthread.so.0
0x00007eff332b4040 0x00007eff332c8c80 Yes (*) /lib/x86_64-linux-gnu/libcrypt.so.1
0x00007eff3322e670 0x00007eff3329423d Yes /usr/local/openresty/luajit/lib/libluajit-5.1.so.2
0x00007eff331b62c0 0x00007eff33205e5d Yes (*) /usr/local/openresty/pcre/lib/libpcre.so.1
0x00007eff3313d750 0x00007eff3318974a Yes (*) /usr/local/openresty/openssl111/lib/libssl.so.1.1
0x00007eff32eca000 0x00007eff3305b260 Yes (*) /usr/local/openresty/openssl111/lib/libcrypto.so.1.1
0x00007eff32e36280 0x00007eff32e4765b Yes (*) /usr/local/openresty/zlib/lib/libz.so.1
0x00007eff32c63630 0x00007eff32dd84bd Yes /lib/x86_64-linux-gnu/libc.so.6
0x00007eff3331f100 0x00007eff33341684 Yes /lib64/ld-linux-x86-64.so.2
0x00007eff32aff3c0 0x00007eff32ba5fa8 Yes /lib/x86_64-linux-gnu/libm.so.6
0x00007eff32ada5e0 0x00007eff32aeb055 Yes (*) /lib/x86_64-linux-gnu/libgcc_s.so.1
0x00007eff32ac15c0 0x00007eff32ac7a1c Yes /lib/x86_64-linux-gnu/libnss_files.so.2
0x00007eff333173e0 0x00007eff3331864c Yes /usr/local/openresty/nginx/modules/ngx_http_geoip2_module.so
0x00007eff32ab2300 0x00007eff32ab4e5c Yes (*) /lib/x86_64-linux-gnu/libmaxminddb.so.0
0x00007eff271766e0 0x00007eff2717a0c1 Yes /usr/local/openresty/lualib/cjson.so
0x00007eff270fefa0 0x00007eff27123300 Yes (*) /lib/x86_64-linux-gnu/libnss_systemd.so.2
(*): Shared library is missing debugging information.
Core was generated by `nginx: worker process '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 ngx_palloc_large (pool=0x55d85c102a00, size=<optimized out>) at src/core/ngx_palloc.c:228
228 src/core/ngx_palloc.c: No such file or directory.
(gdb) bt
#0 ngx_palloc_large (pool=0x55d85c102a00, size=<optimized out>) at src/core/ngx_palloc.c:228
#1 0x000055d85a2614ff in ngx_palloc (pool=<optimized out>, size=<optimized out>) at src/core/ngx_palloc.c:131
#2 0x000055d85a2d8a75 in ngx_http_v2_add_header (h2c=h2c@entry=0x55d85c18a500, header=header@entry=0x55d85c18a5a0) at src/http/v2/ngx_http_v2_table.c:212
#3 0x000055d85a2d6c4b in ngx_http_v2_state_process_header (h2c=0x55d85c18a500,
pos=0x55d85bf57d56 "A\217%\225ȕ\350\067r\364\032)5K\220\364\377S\260I|\245\211\323M\037C\256\272\fA\244ǩ\217\063\246\232?ߚh\372\035u\320b\r&=Ly\246\217\276\320\001w\376\276X\371\373", <incomplete sequence \355>,
end=0x55d85bf57e70 "\177\222\235)\255\027\030cǏ\v\216\241\321!\252_ϟ?@\212AH\264\245I'Y\006I\177\207%\207B\026A\222_@\212AH\264\245I'Z\223\310_\203!\354G@\212AH\264\245I'ZB\241?\204-5\247\327s\223\235)\255\027\030cǏ\v\216\241\321!\252_ϟ,\177P\222\233٫\372RB\313@\322_\245#\263\351OhL\237Q\234\">\213W\337h") at src/http/v2/ngx_http_v2.c:1778
#4 0x000055d85a2d736f in ngx_http_v2_state_field_huff (h2c=<optimized out>, pos=<optimized out>, end=<optimized out>) at src/http/v2/ngx_http_v2.c:1617
#5 0x000055d85a2d7608 in ngx_http_v2_state_field_len (h2c=h2c@entry=0x55d85c18a500, pos=<optimized out>,
end=end@entry=0x55d85bf57e70 "\177\222\235)\255\027\030cǏ\v\216\241\321!\252_ϟ?@\212AH\264\245I'Y\006I\177\207%\207B\026A\222_@\212AH\264\245I'Z\223\310_\203!\354G@\212AH\264\245I'ZB\241?\204-5\247\327s\223\235)\255\027\030cǏ\v\216\241\321!\252_ϟ,\177P\222\233٫\372RB\313@\322_\245#\263\351OhL\237Q\234\">\213W\337h") at src/http/v2/ngx_http_v2.c:1575
#6 0x000055d85a2d7886 in ngx_http_v2_state_header_block (h2c=0x55d85c18a500, pos=<optimized out>,
end=0x55d85bf57e70 "\177\222\235)\255\027\030cǏ\v\216\241\321!\252_ϟ?@\212AH\264\245I'Y\006I\177\207%\207B\026A\222_@\212AH\264\245I'Z\223\310_\203!\354G@\212AH\264\245I'ZB\241?\204-5\247\327s\223\235)\255\027\030cǏ\v\216\241\321!\252_ϟ,\177P\222\233٫\372RB\313@\322_\245#\263\351OhL\237Q\234\">\213W\337h") at src/http/v2/ngx_http_v2.c:1491
#7 0x000055d85a2d5145 in ngx_http_v2_read_handler (rev=0x7eff26bb8d70) at src/http/v2/ngx_http_v2.c:432
#8 0x000055d85a28a596 in ngx_epoll_process_events (cycle=<optimized out>, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:901
#9 0x000055d85a280978 in ngx_process_events_and_timers (cycle=cycle@entry=0x55d85bd276d0) at src/event/ngx_event.c:258
#10 0x000055d85a288690 in ngx_worker_process_cycle (cycle=cycle@entry=0x55d85bd276d0, data=data@entry=0x5) at src/os/unix/ngx_process_cycle.c:793
#11 0x000055d85a286f8d in ngx_spawn_process (cycle=cycle@entry=0x55d85bd276d0, proc=proc@entry=0x55d85a288610 <ngx_worker_process_cycle>, data=data@entry=0x5, name=name@entry=0x55d85a3a3ceb "worker process", respawn=respawn@entry=-3)
at src/os/unix/ngx_process.c:199
#12 0x000055d85a288ba4 in ngx_start_worker_processes (cycle=cycle@entry=0x55d85bd276d0, n=8, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:382
#13 0x000055d85a289442 in ngx_master_process_cycle (cycle=0x55d85bd276d0) at src/os/unix/ngx_process_cycle.c:135
#14 0x000055d85a25ed7a in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:387
f 0
(gdb) info local
p = 0x55d85bfe8990
n = 2
large = 0x9f5d259b3eb91b2
(gdb) p *pool
$38 = {d = {last = 0x55d85c102bfc "\330U", end = 0x55d85c102c00 "\020\002", next = 0x55d85c002ef0, failed = 1}, max = 432, current = 0x55d85c102a00, chain = 0x0, large = 0x55d85c0030d8, cleanup = 0x55d85c002fa0, log = 0x55d85c102a60}
f 2
#2 0x000055d85a2d8a75 in ngx_http_v2_add_header (h2c=h2c@entry=0x55d85c18a500, header=header@entry=0x55d85c18a5a0) at src/http/v2/ngx_http_v2_table.c:212
$12 = {connection = 0x7eff26d3e180, http_connection = 0x55d85c102ac0, total_bytes = 130, payload_bytes = 0, processing = 1, frames = 0, idle = 3, new_streams = 1, refused_streams = 0, priority_limit = 256, send_window = 65535,
recv_window = 2147483647, init_window = 65535, frame_size = 16384, waiting = {prev = 0x55d85c18a570, next = 0x55d85c18a570}, state = {sid = 1, length = 282, padding = 0, flags = 5, incomplete = 0, keep_pool = 1, parse_name = 0,
parse_value = 0, index = 1, header = {name = {len = 5, data = 0x55d85a3abcbe ":path"}, value = {len = 165,
data = 0x55d85c0ea340 "***SENSITIVE INFO (url)***"}}, header_limit = 32576,
field_state = 0 '\000', field_start = 0x55d85c0ea340 "***SENSITIVE INFO (url)***",
field_end = 0x55d85c0ea3e5 "", field_rest = 0, pool = 0x55d85c0ea2f0, stream = 0x55d85bfc9ee0, buffer = '\000' <repeats 15 times>, buffer_used = 0, handler = 0x55d85a2d77e0 <ngx_http_v2_state_header_block>}, hpack = {
entries = 0x55d85c204dc0, added = 0, deleted = 0, reused = 0, allocated = 64, size = 4096, free = 4096, storage = 0x0, pos = 0x0}, pool = 0x55d85c032990, free_frames = 0x0, free_fake_connections = 0x0,
streams_index = 0x55d85c002fb8, last_out = 0x0, dependencies = {prev = 0x55d85bfc0398, next = 0x55d85bfc0398}, closed = {prev = 0x55d85c18a698, next = 0x55d85c18a698}, closed_nodes = 0, last_sid = 1, lingering_time = 0,
settings_ack = 1, table_update = 1, blocked = 1, goaway = 0, fp_fingerprinted = 0, fp_settings = {len = 35, data = 0x55d85c0030b8 "\001"}, fp_windowupdate = 0, fp_priorities = {len = 4, data = 0x55d85bfc02f0 "\001"},
fp_pseudoheaders = {len = 2, data = 0x55d85bfc0310 "ce\374[\330U"}, fp_str = {len = 0, data = 0x0}}
(gdb) p header
$13 = (ngx_http_v2_header_t *) 0x55d85c18a5a0
(gdb) p header->name
$14 = {len = 5, data = 0x55d85a3abcbe ":path"}
(gdb) p header->value
$15 = {len = 165, data = 0x55d85c0ea340 "***SENSITIVE INFO (url)***"}
(gdb) p h2c->hpack
$17 = {entries = 0x55d85c204dc0, added = 0, deleted = 0, reused = 0, allocated = 64, size = 4096, free = 4096, storage = 0x0, pos = 0x0}
(gdb) p **h2c->hpack->entries
$20 = {name = {len = 94387745532352, data = 0x55d85bd08010 "\a"}, value = {len = 0, data = 0x0}}
(gdb) p *h2c->connection
$24 = {data = 0x55d85c18a500, read = 0x7eff26bb8d70, write = 0x7eff26a37d70, fd = 49, recv = 0x55d85a28fc60 <ngx_ssl_recv>, send = 0x55d85a290100 <ngx_ssl_write>, recv_chain = 0x55d85a290040 <ngx_ssl_recv_chain>,
send_chain = 0x55d85a290530 <ngx_ssl_send_chain>, listening = 0x55d85bd27f10, sent = 58, log = 0x55d85c102a60, pool = 0x55d85c102a00, type = 1, sockaddr = 0x55d85c102a50, socklen = 16, addr_text = {len = 12,
data = 0x55d85c102ab0 "84.*IP****"}, proxy_protocol = 0x0, ssl = 0x55d85c102b18, udp = 0x0, local_sockaddr = 0x55d85bf44430, local_socklen = 16, buffer = 0x0, queue = {prev = 0x7eff26d38748, next = 0x7eff26d38fb8},
number = 737234, start_time = 84136899431, requests = 1, buffered = 0, log_error = 2, timedout = 0, error = 0, destroyed = 0, pipeline = 0, idle = 1, reusable = 0, close = 0, shared = 0, sendfile = 0, sndlowat = 0, tcp_nodelay = 1,
tcp_nopush = 0, need_last_buf = 0, need_flush_buf = 0, sendfile_task = 0x0}
(gdb) p *h2c->connection->pool
$25 = {d = {last = 0x55d85c102bfc "\330U", end = 0x55d85c102c00 "\020\002", next = 0x55d85c002ef0, failed = 1}, max = 432, current = 0x55d85c102a00, chain = 0x0, large = 0x55d85c0030d8, cleanup = 0x55d85c002fa0, log = 0x55d85c102a60}
(gdb)
(gdb) f 3
#3 0x000055d85a2d6c4b in ngx_http_v2_state_process_header (h2c=0x55d85c18a500,
pos=0x55d85bf57d56 "A\217%\225ȕ\350\067r\364\032)5K\220\364\377S\260I|\245\211\323M\037C\256\272\fA\244ǩ\217\063\246\232?ߚh\372\035u\320b\r&=Ly\246\217\276\320\001w\376\276X\371\373", <incomplete sequence \355>,
end=0x55d85bf57e70 "\177\222\235)\255\027\030cǏ\v\216\241\321!\252_ϟ?@\212AH\264\245I'Y\006I\177\207%\207B\026A\222_@\212AH\264\245I'Z\223\310_\203!\354G@\212AH\264\245I'ZB\241?\204-5\247\327s\223\235)\255\027\030cǏ\v\216\241\321!\252_ϟ,\177P\222\233٫\372RB\313@\322_\245#\263\351OhL\237Q\234\">\213W\337h") at src/http/v2/ngx_http_v2.c:1778
1778 src/http/v2/ngx_http_v2.c: No such file or directory.
(gdb) p pos
$39 = (u_char *) 0x55d85bf57d56 "A\217%\225ȕ\350\067r\364\032)5K\220\364\377S\260I|\245\211\323M\037C\256\272\fA\244ǩ\217\063\246\232?ߚh\372\035u\320b\r&=Ly\246\217\276\320\001w\376\276X\371\373", <incomplete sequence \355>
(gdb) p end
$42 = (u_char *) 0x55d85bf57e70 "\177\222\235)\255\027\030cǏ\v\216\241\321!\252_ϟ?@\212AH\264\245I'Y\006I\177\207%\207B\026A\222_@\212AH\264\245I'Z\223\310_\203!\354G@\212AH\264\245I'ZB\241?\204-5\247\327s\223\235)\255\027\030cǏ\v\216\241\321!\252_ϟ,\177P\222\233٫\372RB\313@\322_\245#\263\351OhL\237Q\234\">\213W\337h"
(gdb) p *h2c->connection->log
$47 = {log_level = 4, file = 0x55d85bd27a70, connection = 737234, disk_full_time = 0, handler = 0x55d85a2a1a30 <ngx_http_log_error>, data = 0x55d85c102b00, writer = 0x0, wdata = 0x0,
action = 0x55d85a3aaaec "processing HTTP/2 connection", next = 0x0}
(gdb) p **h2c->streams_index
$51 = {id = 1, index = 0x0, parent = 0xffffffffffffffff, queue = {prev = 0x55d85c18a688, next = 0x55d85c18a688}, children = {prev = 0x55d85bfc03a8, next = 0x55d85bfc03a8}, reuse = {prev = 0x0, next = 0x0}, rank = 1, weight = 16,
rel_weight = 0.0625, stream = 0x55d85bfc9ee0}
(gdb) p *ngx_cycle
$57 = {conf_ctx = 0x55d85bd28978, pool = 0x55d85bd27680, log = 0x55d85bd276e8, new_log = {log_level = 4, file = 0x55d85bd27a70, connection = 0, disk_full_time = 0, handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0,
next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x7eff26d384b0, free_connection_n = 16278, modules = 0x55d85bd290e0, modules_n = 107, modules_used = 1, reusable_connections_queue = {prev = 0x7eff26d39468,
next = 0x7eff26d3c528}, reusable_connections_n = 93, connections_reuse_time = 0, listening = {elts = 0x55d85bd27de8, nelts = 7, size = 296, nalloc = 10, pool = 0x55d85bd27680}, paths = {elts = 0x55d85bd27a08, nelts = 6, size = 8,
nalloc = 10, pool = 0x55d85bd27680}, config_dump = {elts = 0x55d85bd27a58, nelts = 0, size = 24, nalloc = 1, pool = 0x55d85bd27680}, config_dump_rbtree = {root = 0x55d85bd27820, sentinel = 0x55d85bd27820,
insert = 0x55d85a2664c0 <ngx_str_rbtree_insert_value>}, config_dump_sentinel = {key = 0, left = 0x0, right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, open_files = {last = 0x55d85bd27850, part = {
elts = 0x55d85bd27a70, nelts = 1, next = 0x0}, size = 40, nalloc = 20, pool = 0x55d85bd27680}, shared_memory = {last = 0x55d85bd4c108, part = {elts = 0x55d85bd27d90, nelts = 1, next = 0x55d85bd4bfd0}, size = 88, nalloc = 1,
pool = 0x55d85bd27680}, connection_n = 16384, files_n = 0, connections = 0x7eff26d37010, read_events = 0x7eff26bb6010, write_events = 0x7eff26a35010, old_cycle = 0x0, conf_file = {len = 42,
data = 0x55d85bd279ba "/usr/local/openresty/nginx/conf/nginx.conf"}, conf_param = {len = 29, data = 0x55d85bd279e5 "daemon on; master_process on;\032\063\377~"}, conf_prefix = {len = 32,
data = 0x55d85bd27970 "/usr/local/openresty/nginx/conf//usr/local/openresty/nginx/logs/error.log"}, prefix = {len = 27, data = 0x55d85bd27990 "/usr/local/openresty/nginx/logs/error.log"}, error_log = {len = 14,
data = 0x55d85bd279ab "logs/error.log"}, lock_file = {len = 43, data = 0x55d85bf446fc "/usr/local/openresty/nginx/logs/nginx.lock.accept"}, hostname = {len = 24, data = 0x55d85bd290c8 "*************\200?Z\330U"},
intercept_error_log_handler = 0x0, intercept_error_log_data = 0x0, entered_logger = 0}
seems there're a buffer overflow in ja3, let me try to figure out.
If you need help in testing / debugging I can build & deploy in 15 minutes to see if a fix works. I also shared a private github repo with the last core & binary & debug symbols.
I had time to check up on this. I created a patch where I check allocation success & etc and its working just fine now, no segfaults.
I think this might have been the main issue in ngx_http_v2.c:
if (!h2c->fp_fingerprinted && h2c->fp_settings.len < 32) {
h2c->fp_settings.data[h2c->fp_settings.len] = (uint8_t)id;
*(uint32_t*)(h2c->fp_settings.data + h2c->fp_settings.len + 1) = (uint32_t)value;
h2c->fp_settings.len += 5;
}
I am not familiar with ngx internal data structures so I checked every allocation and variables to make sure there is no issue with them, but might not needed at all at the end. So please somehow keep only the necessary checks you know its needed.
nginx-patch.tar.gz