observing nginx core files when webserver running on longrun with websocket open
prasu346 opened this issue · comments
In our environment, we are using Openresty packge for webserver needs. Recently we have upgraded openresty to 1.21.3 and observing nginx core files when webserver running on long run with websocket open
nginx -V
nginx version: openresty/1.21.4.3
built by clang 11.0.1 (git@github3.cisco.com:llvm-dev/llvm-11.git 317296e2b4bebedb01c87904614fca39dac5fdf1)
built with SSL 1.1.1u.
TLS SNI support enabled
uname -a
x86_64 GNU/Linux
Below is the crash decode of the core file
(gdb) bt
#0 0x0000738ba04d2592 in trace_abort (J=0x40f94560)
luajit2-2.1-20190912/src/lj_trace.c:585
#1 trace_state (L=, dummy=, ud=0x40f94560)
luajit2-2.1-20190912/src/lj_trace.c:710
#2 0x0000738ba04d5746 in lj_vm_cpcall () at lj_vm.s:1179
#3 0x0000738ba04d220e in lj_trace_ins (J=J@entry=0x40f94560, pc=pc@entry=0x40754f88)
luajit2-2.1-20190912/src/lj_trace.c:730
#4 0x0000738ba04a221d in lj_dispatch_ins (L=0x40bcf840, pc=0x40754f8c)
luajit2-2.1-20190912/src/lj_dispatch.c:424
#5 0x0000738ba04d6d41 in lj_vm_inshook () lj_vm.s:2348
#6 0x000055f73c34fbee in ngx_http_lua_run_thread (L=, L@entry=0x40f94378, r=, r@entry=0x55f73e518ae0, ctx=, ctx@entry=0x55f73e588710, nrets=0) at ../ngx_lua-0.10.25/src/ngx_http_lua_util.c:1184
#7 0x000055f73c357837 in ngx_http_lua_rewrite_by_chunk (L=0x40f94378, r=0x55f73e518ae0)
at ../ngx_lua-0.10.25/src/ngx_http_lua_rewriteby.c:337
#8 0x000055f73c3573d4 in ngx_http_lua_rewrite_handler (r=0x55f73e518ae0) at ../ngx_lua-0.10.25/src/ngx_http_lua_rewriteby.c:162
#9 0x000055f73c2acbcf in ngx_http_core_rewrite_phase (r=0x55f73e518ae0, ph=0x55f73e428fa0) at src/http/ngx_http_core_module.c:939
#10 0x000055f73c2aca18 in ngx_http_core_run_phases (r=0x55f73e518ae0) at src/http/ngx_http_core_module.c:885
#11 ngx_http_handler (r=r@entry=0x55f73e518ae0) at src/http/ngx_http_core_module.c:868
#12 0x000055f73c2b822d in ngx_http_process_request (r=r@entry=0x55f73e518ae0) at src/http/ngx_http_request.c:2120
#13 0x000055f73c2ba0d7 in ngx_http_process_request_headers (rev=, rev@entry=0x55f73e4890f0)
at src/http/ngx_http_request.c:1498
#14 0x000055f73c2b9be4 in ngx_http_process_request_line (rev=0x55f73e4890f0) at src/http/ngx_http_request.c:1165
#15 0x000055f73c29c379 in ngx_epoll_process_events (cycle=0x55f73de2f5d0, timer=, flags=1)
at src/event/modules/ngx_epoll_module.c:901
#16 0x000055f73c28db53 in ngx_process_events_and_timers (cycle=cycle@entry=0x55f73de2f5d0) at src/event/ngx_event.c:258
#17 0x000055f73c29a2b0 in ngx_worker_process_cycle (cycle=0x55f73de2f5d0, data=) at src/os/unix/ngx_process_cycle.c:793
#18 0x000055f73c297bcb in ngx_spawn_process (cycle=, cycle@entry=0x55f73de2f5d0, proc=0x400, data=0x0,
name=0x55f73c3c30d2 "worker process", respawn=0) at src/os/unix/ngx_process.c:199
#19 0x000055f73c29919a in ngx_reap_children (cycle=) at src/os/unix/ngx_process_cycle.c:665
#20 ngx_master_process_cycle (cycle=, cycle@entry=0x55f73de2f5d0) at src/os/unix/ngx_process_cycle.c:180
#21 0x000055f73c265b63 in main (argc=, argv=) at src/core/nginx.c:386
(gdb) frame 7
#7 0x000055f73c357837 in ngx_http_lua_rewrite_by_chunk (L=0x40f94378, r=0x55f73e518ae0)
at ../ngx_lua-0.10.25/src/ngx_http_lua_rewriteby.c:337
decode frame 6
(gdb) p *r@entry
$5 = {signature = 1347703880, connection = 0x55f73e443b40, ctx = 0x55f73e519540, main_conf = 0x55f73de33360,
srv_conf = 0x55f73dfd6460, loc_conf = 0x55f73e0b7c60, read_event_handler = 0x55f73c2b86b0 <ngx_http_block_reading>,
write_event_handler = 0x55f73c2aca40 <ngx_http_core_run_phases>, cache = 0x0, upstream = 0x0, upstream_states = 0x0,
pool = 0x55f73e518a90, header_in = 0x55f73e4be9f0, headers_in = {headers = {last = 0x55f73e518b50, part = {elts = 0x55f73e588350,
nelts = 8, next = 0x0}, size = 48, nalloc = 20, pool = 0x55f73e518a90}, host = 0x55f73e588380, connection = 0x0,
if_modified_since = 0x0, if_unmodified_since = 0x0, if_match = 0x0, if_none_match = 0x0, user_agent = 0x0, referer = 0x0,
content_length = 0x0, content_range = 0x0, content_type = 0x0, range = 0x0, if_range = 0x0, transfer_encoding = 0x0, te = 0x0,
expect = 0x0, upgrade = 0x55f73e5883e0, accept_encoding = 0x0, via = 0x0, authorization = 0x0, keep_alive = 0x0,
x_forwarded_for = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, x_real_ip = 0x0, depth = 0x0, destination = 0x0,
overwrite = 0x0, date = 0x0, user = {len = 0, data = 0x0}, passwd = {len = 0, data = 0x0}, cookies = {elts = 0x55f73e519740,
nelts = 1, size = 8, nalloc = 1, pool = 0x55f73e518a90}, server = {len = 4,
data = 0x55f73e56796d "fd0a:9b09:1f7:1:92eb:50ff:fe63:7780"}, content_length_n = -1, keep_alive_n = -1, connection_type = 0,
chunked = 0, msie = 0, msie6 = 0, opera = 0, gecko = 0, chrome = 0, safari = 0, konqueror = 0}, headers_out = {headers = {
last = 0x55f73e518cf0, part = {elts = 0x55f73e5190c0, nelts = 0, next = 0x0}, size = 48, nalloc = 20, pool = 0x55f73e518a90},
trailers = {last = 0x55f73e518d28, part = {elts = 0x55f73e519480, nelts = 0, next = 0x0}, size = 48, nalloc = 4,
pool = 0x55f73e518a90}, status = 0, status_line = {len = 0, data = 0x0}, server = 0x0, date = 0x0, content_length = 0x0,
content_encoding = 0x0, location = 0x0, refresh = 0x0, last_modified = 0x0, content_range = 0x0, accept_ranges = 0x0,
www_authenticate = 0x0, expires = 0x0, etag = 0x0, override_charset = 0x0, content_type_len = 0, content_type = {len = 0,
data = 0x0}, charset = {len = 0, data = 0x0}, content_type_lowcase = 0x0, content_type_hash = 0, cache_control = {elts = 0x0,
nelts = 0, size = 0, nalloc = 0, pool = 0x0}, link = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},
content_length_n = -1, content_offset = 0, date_time = 0, last_modified_time = -1}, request_body = 0x55f73e519a10,
lingering_time = 0, start_sec = 1710111079, start_msec = 400, method = 2, http_version = 1001, request_line = {len = 44,
data = 0x55f73e5677c0 "GET /example_loc HTTP/1.1\r\nCookie"}, uri = {len = 31,
data = 0x55f73e5677c4 "/example_loc HTTP/1.1\r\nCookie"}, args = {len = 0, data = 0x0}, exten = {len = 0,
data = 0x0}, unparsed_uri = {len = 31, data = 0x55f73e5677c4 "/example_loc HTTP/1.1\r\nCookie"}, method_name = {
len = 3, data = 0x55f73e5677c0 "GET /example_loc HTTP/1.1\r\nCookie"}, http_protocol = {len = 8,
data = 0x55f73e5677e4 "HTTP/1.1\r\nCookie"}, schema = {len = 0, data = 0x0}, out = 0x0, main = 0x55f73e518ae0, parent = 0x0,
postponed = 0x0, post_subrequest = 0x0, posted_requests = 0x0, phase_handler = 6,
content_handler = 0x55f73c2fc540 <ngx_http_proxy_handler>, access_code = 0, variables = 0x55f73e587fd0, ncaptures = 0,
captures = 0x55f73e5198a0, captures_data = 0x0, limit_rate = 0, limit_rate_after = 0, header_size = 0, request_length = 724, err_status = 0, http_connection = 0x55f73e2dcb90, stream = 0x0, log_handler = 0x55f73c2ba830 <ngx_http_log_error_handler>, cleanup = 0x0, count = 1, subrequests = 51, blocked = 0, aio = 0, http_state = 2, complex_uri = 0, quoted_uri = 0, plus_in_uri = 0, empty_path_in_uri = 0, invalid_header = 0, add_uri_to_alias = 0, valid_location = 1, valid_unparsed_uri = 1, uri_changed = 0, uri_changes = 11, request_body_in_single_buf = 1, request_body_in_file_only = 0, request_body_in_persistent_file = 1, request_body_in_clean_file = 1, request_body_file_group_access = 0, request_body_file_log_level = 5, request_body_no_buffering = 0, subrequest_in_memory = 0, waited = 0, cached = 0, gzip_tested = 0, gzip_ok = 0, gzip_vary = 0, realloc_captures = 0, proxy = 0, bypass_cache = 0, no_cache = 0, limit_conn_status = 0, limit_req_status = 0, limit_rate_set = 0, limit_rate_after_set = 0, pipeline = 0, chunked = 0, header_only = 0, expect_trailers = 0, keepalive = 1, lingering_close = 0, discard_body = 0, reading_body = 0, internal = 0, error_page = 0, filter_finalize = 0, post_action = 0, request_complete = 0, request_output = 0, header_sent = 0, expect_tested = 0, root_tested = 0, done = 0, logged = 0, buffered = 0, main_filter_need_in_memory = 1, filter_need_in_memory = 0, filter_need_temporary = 0, preserve_body = 0, allow_ranges = 0, subrequest_ranges = 0, single_range = 0, disable_not_modified = 0, stat_reading = 0, stat_writing = 1, stat_processing = 0, background = 0, health_check = 0, state = 0, header_hash = 2446989189541711446, lowcase_index = 21, lowcase_header = "sec-websocket-versionl\000\000\000\000\000\000\000\000\000", header_name_start = 0x55f73e567a92 "\r\ndap://log4shell-generic-T0Q8ewrobSEkh31hecIQ${lower:ten}.w.nessus.org/nessus};PHPSESSID=${jndi:ldap://log4shell-generic-T0Q8ewrobSEkh31hecIQ${lower:ten}.w.nessus.org/nessus};token=${jndi:ldap://log4"..., header_name_end = 0x55f73e567a8c "", header_start = 0x55f73e567a8e "13", header_end = 0x55f73e567a92 "\r\ndap://log4shell-generic-T0Q8ewrobSEkh31hecIQ${lower:ten}.w.nessus.org/nessus};PHPSESSID=${jndi:ldap://log4shell-generic-T0Q8ewrobSEkh31hecIQ${lower:ten}.w.nessus.org/nessus};token=${jndi:ldap://log4"..., uri_start = 0x55f73e5677c4 "/example_loc HTTP/1.1\r\nCookie", uri_end = 0x55f73e5677e3 " HTTP/1.1\r\nCookie", uri_ext = 0x0, args_start = 0x0, request_start = 0x55f73e5677c0 "GET /example_loc HTTP/1.1\r\nCookie", request_end = 0x55f73e5677ec "\r\nCookie", method_end = 0x55f73e5677c2 "T /example_loc HTTP/1.1\r\nCookie", schema_start = 0x0, schema_end = 0x0, host_start = 0x0, host_end = 0x0, port_start = 0x0, port_end = 0x0, http_minor = 1, http_major = 1}
Debug logs when the device core file seen
2024/03/18 07:40:21 [debug] 29113#0: *2591 rewrite phase: 6
2024/03/18 07:40:21 [debug] 29113#0: *2591 lua rewrite handler, uri:"/example_loc" c:1
2024/03/18 07:40:21 [debug] 29113#0: *2591 code cache lookup (key='rewrite_by_lua_nhli_65a67b58b1ac094eac31898a087c086c', ref=4)
2024/03/18 07:40:21 [debug] 29113#0: *2591 code cache hit (key='rewrite_by_lua_nhli_65a67b58b1ac094eac31898a087c086c', ref=4)
2024/03/18 07:40:21 [debug] 29113#0: *2591 lua reset ctx
2024/03/18 07:40:21 [debug] 29113#0: *2591 http lua finalize threads
2024/03/18 07:40:21 [debug] 29113#0: *2591 add cleanup: 000060A83EA03C20
2024/03/18 07:40:21 [debug] 29113#0: *2591 lua run thread, top:0 c:1
2024/03/18 07:40:21 [warn] 29113#0: *2591 using uninitialized "validation" variable, client: fd0a:9b09:1f7:1:9143:7c78:7cf2:3bb6, server: , request: "GET /example_loc HTTP/1.1", host: "[fd0a:9b09:1f7:1:92eb:50ff:fe63:7780]"
2024/03/18 07:40:21 [debug] 29113#0: *2591 lua start to read buffered request body
2024/03/18 07:40:21 [debug] 29113#0: *2591 lua req body post read, c:2
2024/03/18 07:40:21 [debug] 29113#0: *2591 lua has read buffered request body in a single run
2024/03/18 07:40:21 [debug] 29113#0: *2591 parse header: "Cookie: Auth=56e72649dd57dcab27f5b3a87433f8ca25e4177c7fba19b91b46683b021e6d93c45b81df9b07272f228376011f943338cddce10a70b9ea030c93400c557e57ce841ee1086215449c59768b3545beb85c814b13127ce06b61d42385b23d40deb965f133eed90aa065f32496077281642d460931e1dc916f0ff54f5b010385a313:804974de5a0d26fa02182578bd904d5aadb43a30b4ed075c432b80c249555c3f; Path=/example_loc1; Domain=[fd0a:9b09:1f7:1:92eb:50ff:fe63:7780]; ; Secure; HttpOnly; SameSite=Lax"
2024/03/18 07:40:21 [debug] 26875#0: epoll add event: fd:8 op:1 ev:00002001
2024/03/18 07:40:21 [debug] 26875#0: epoll add event: fd:9 op:1 ev:00002001
2024/03/18 07:40:21 [debug] 26875#0: epoll add event: fd:10 op:1 ev:00002001
2024/03/18 07:40:21 [debug] 26875#0: epoll add event: fd:6 op:1 ev:00002001
2024/03/18 07:40:23 [debug] 26875#0: accept on [::]:443, ready: 0
2024/03/18 07:40:23 [debug] 26875#0: posix_memalign: 000060A83E64A310:512 @16
2024/03/18 07:40:23 [debug] 26875#0: *2593 accept: [fd0a:9b09:1f7:1:9143:7c78:7cf2:3bb6]:60174 fd:27
2024/03/18 07:40:23 [debug] 26875#0: malloc: 000060A83EC5CCF0:16
2024/03/18 07:40:23 [debug] 26875#0: *2593 event timer add: 27: 10000:256220612
2024/03/18 07:40:23 [debug] 26875#0: *2593 reusable connection: 1
2024/03/18 07:40:23 [debug] 26875#0: *2593 epoll add event: fd:27 op:1 ev:80002001
2024/03/18 07:40:23 [debug] 26875#0: *2593 http check ssl handshake
2024/03/18 07:40:23 [debug] 26875#0: *2593 http recv(): 1
2024/03/18 07:40:23 [debug] 26875#0: *2593 https ssl handshake: 0x16
2024/03/18 07:40:23 [debug] 26875#0: *2593 tcp_nodelay
2024/03/18 07:40:23 [debug] 26875#0: *2593 reusable connection: 0
2024/03/18 07:40:23 [debug] 26875#0: *2593 SSL server name: null
2024/03/18 07:40:23 [debug] 26875#0: *2593 ssl cert: connection reusable: 0
2024/03/18 07:40:23 [debug] 26875#0: *2594 code cache lookup (key='ssl_certificate_by_lua_nhli_52c41e130f78cccaaed4c6149e003cf3', ref=-1)
2024/03/18 07:40:23 [debug] 26875#0: *2594 code cache miss (key='ssl_certificate_by_lua_nhli_52c41e130f78cccaaed4c6149e003cf3', ref=-1)
and when i decode frame
Thanks for your cooperation.
would you please update Luajit to the latest version？
updated luajit to luajit2-2.1-2023100 version but ended up with below traceback when i am starting my service on
x86_64.
Thread 1 (Thread 0x7ed44bd0a780 (LWP 3815)):
#0 0x00007ed44c078469 in lj_vm_cpcall () at lj_vm.s:1183
#1 0x00007ed44c06c0ba in lua_newstate (allocf=0x7ed44c009790 <lj_alloc_f>, allocf@entry=0x4d50, allocd=0x7ed44bcea010, allocd@entry=0x0) at luajit2-2.1-20231006/src/lj_state.c:276
#2 0x00007ed44bffb50a in luaL_newstate () at contrib/luajit/_gen_extract-cat9k_universalk9-x86_64_cge7/openresty-luajit2-2.1-20231006/src/lib_aux.c:378
#3 0x00005bb807cf772a in main (argc=5, argv=0x7ffea90477d8) at luajit2-2.1-20231006/src/luajit.c:574
Is there any stable recommended version for openresty and luajit?
Hi,
Can anyone please help me to understand why am i seeing above traceback from luajit.
There seems to be some bugs in your lua code in rewrite_by_lua, rewrite_by_lua_block or rewrite_by_lua_file.
0x000055f73c357837 in ngx_http_lua_rewrite_by_chunk (L=0x40f94378, r=0x55f73e518ae0)
In addition please check the variable "validation" in conf file.
2024/03/18 07:40:21 [warn] 29113#0: *2591 using uninitialized "validation" variable,
Hi,
I have upgraded my luajit to latest luajit2-2.1-20231117.
after upgarde, my luajit is crashing when i try to run below simple code on x86 platform.
Below is the my code and traceback
code snippet:
LD_LIBRARY_PATH=$library_path
$LUA -e "local log = require('log') log.force()"
mkdir -p $ROOT/$xyz
echo "Applying $xyz/$name($ver) from $mount"
LD_LIBRARY_PATH=$library_path
$LUA -l Target
-e "applyDatabase('$name', '$ver', '$xyz', true, '$0')
error logs:
3470 Segmentation fault (core dumped)
LD_LIBRARY_PATH=$library_path $LUA -e "local log = require('log') log.force()"
[ 30.907511] localhost : /usr/binos/conf/example_init.sh: line 152: 3473 Segmentation fault (core dumped) LD_LIBRARY_PATH=$library_path $LUA -l Target -e "applyDatabase('$name', '$ver', '$abc', true, '$0')"
traceback:
#0 0x00007f0e7baec0e9 in lj_vm_cpcall () at x86_64_cge7/lj_vm.s:1183
#1 0x00007f0e7badf436 in lua_newstate (allocf=0x7f0e7ba6d480 <mem_alloc>, allocd=allocd@entry=0x0)
at x86_64_cge7/openresty-luajit2-2.1-20231117/src/lj_state.c:276
#2 0x00007f0e7ba6d40c in luaL_newstate ()
at x86_64_cge7/openresty-luajit2-2.1-20231117/src/lib_aux.c:356
#3 0x0000642960295ada in main (argc=5, argv=0x7ffec0d6e478)
at x86_64_cge7/openresty-luajit2-2.1-20231117/src/luajit.c:574
crash is happening aftre upgrade to latest luajit, with old luajit it is working without any issue.