[CRASH] Redis 7.2.4 crashed by signal: 6, si_code: -6
doxg opened this issue · comments
Greetings,
I am currently utilizing the redis-server for the purpose of caching within my application. The workflow involves the reception of real-time data on a per-second basis, wherein I proceed to store these values in Redis utilizing a key:list format, with a limit of 100 elements per list. Subsequently, this data is transferred to a database. Upon completion of this transfer, all corresponding keys within Redis are removed. Then, the cycle repeats. The system have been running two days before the redis container exited with code (134).
Crash report
=== REDIS BUG REPORT START: Cut & paste starting from here ===
9:M 10 Apr 2024 06:18:32.671 # Redis 7.2.4 crashed by signal: 6, si_code: -6
9:M 10 Apr 2024 06:18:32.671 # Crashed running the instruction at: 0x7a3700bdf9fc
------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7a3700bdf9fc]
Backtrace:
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7a3700b8b520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7a3700bdf9fc]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7a3700b8b476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7a3700b717f3]
/lib/x86_64-linux-gnu/libc.so.6(+0x89676)[0x7a3700bd2676]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x2a)[0x7a3700c7f59a]
/lib/x86_64-linux-gnu/libc.so.6(+0x136566)[0x7a3700c7f566]
/opt/redis-stack/bin/redis-server *:6379(lindexCommand+0x20e)[0x55def2c84e5e]
/opt/redis-stack/bin/redis-server *:6379(server+0x0)[0x55def2f2d8a0]
------ REGISTERS ------
9:M 10 Apr 2024 06:18:32.672 #
RAX:0000000000000000 RBX:00007a3700b45200
RCX:00007a3700bdf9fc RDX:0000000000000006
RDI:0000000000000009 RSI:0000000000000009
RBP:0000000000000009 RSP:00007ffce9538620
R8 :00007ffce95386f0 R9 :0000000000000000
R10:0000000000000008 R11:0000000000000246
R12:0000000000000006 R13:0000000000000016
R14:0000000000000002 R15:0000000000000001
RIP:00007a3700bdf9fc EFL:0000000000000246
CSGSFS:002b000000000033
9:M 10 Apr 2024 06:18:32.672 # (00007ffce953862f) -> 00007ffce9538998
9:M 10 Apr 2024 06:18:32.672 # (00007ffce953862e) -> 00007a3700b44720
9:M 10 Apr 2024 06:18:32.672 # (00007ffce953862d) -> 7d877b152d958d00
9:M 10 Apr 2024 06:18:32.672 # (00007ffce953862c) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce953862b) -> 7d877b152d958d00
9:M 10 Apr 2024 06:18:32.672 # (00007ffce953862a) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce9538629) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce9538628) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce9538627) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce9538626) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce9538625) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce9538624) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce9538623) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce9538622) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce9538621) -> 00007a3700898280
9:M 10 Apr 2024 06:18:32.672 # (00007ffce9538620) -> 00007a3700898280
------ INFO OUTPUT ------
# Server
redis_version:7.2.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:3a23a5ac67fe7008
redis_mode:standalone
os:Linux 6.5.0-27-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.4.0
process_id:9
process_supervised:no
run_id:8db3f0407ebf1b76b5832fc2bc0c0f5f9aadd0d2
tcp_port:6379
server_time_usec:1712729912671594
uptime_in_seconds:82714
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:1453880
executable:/opt/redis-stack/bin/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=*,bind=-::*,port=6379
# Clients
connected_clients:4
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:20504
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
# Memory
used_memory:1665544
used_memory_human:1.59M
used_memory_rss:21901312
used_memory_rss_human:20.89M
used_memory_peak:2183552
used_memory_peak_human:2.08M
used_memory_peak_perc:76.28%
used_memory_overhead:1288528
used_memory_startup:1193088
used_memory_dataset:377016
used_memory_dataset_perc:79.80%
allocator_allocated:1812544
allocator_active:8781824
allocator_resident:15204352
total_system_memory:33403666432
total_system_memory_human:31.11G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:4.85
allocator_frag_bytes:6969280
allocator_rss_ratio:1.73
allocator_rss_bytes:6422528
rss_overhead_ratio:1.44
rss_overhead_bytes:6696960
mem_fragmentation_ratio:13.42
mem_fragmentation_bytes:20269648
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:84488
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1712673421
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_saves:424
rdb_last_cow_size:1003520
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:213
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:10
total_commands_processed:8096910560
instantaneous_ops_per_sec:96674
total_net_input_bytes:408947426582
total_net_output_bytes:406652407660
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:4769.19
instantaneous_output_kbps:4747.82
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:348
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:7972382342
keyspace_misses:53395
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:259
total_forks:424
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:81752319
total_writes_processed:81753030
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:15861
reply_buffer_expands:47311
eventloop_cycles:82124693
eventloop_duration_sum:2887565653
eventloop_duration_cmd_sum:1492045457
instantaneous_eventloop_cycles_per_sec:911
instantaneous_eventloop_duration_usec:54
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:45446aa02e4549c31c33a978069bff00cdf2cf69
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:455.564253
used_cpu_user:2569.047655
used_cpu_sys_children:0.132226
used_cpu_user_children:0.444681
used_cpu_sys_main_thread:452.858630
used_cpu_user_main_thread:2563.903713
# Modules
module:name=search,ver=20813,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
module:name=RedisCompat,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
module:name=redisgears_2,ver=20020,api=1,filters=0,usedby=[],using=[],options=[]
module:name=timeseries,ver=11012,api=1,filters=0,usedby=[],using=[],options=[]
module:name=ReJSON,ver=20610,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=bf,ver=20612,api=1,filters=0,usedby=[],using=[],options=[]
# Commandstats
cmdstat_keys:calls=43722553,usec=323293037,usec_per_call=7.39,rejected_calls=0,failed_calls=0
cmdstat_lindex:calls=7972388189,usec=657748416,usec_per_call=0.08,rejected_calls=0,failed_calls=0
cmdstat_set:calls=1776,usec=2097,usec_per_call=1.18,rejected_calls=0,failed_calls=0
cmdstat_rpush:calls=4654975,usec=1010346,usec_per_call=0.22,rejected_calls=0,failed_calls=0
cmdstat_multi:calls=38024730,usec=2400862,usec_per_call=0.06,rejected_calls=0,failed_calls=0
cmdstat_flushdb:calls=1,usec=69,usec_per_call=69.00,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=1,usec=5,usec_per_call=5.00,rejected_calls=0,failed_calls=0
cmdstat_lrange:calls=47105,usec=109866,usec_per_call=2.33,rejected_calls=0,failed_calls=0
cmdstat_info:calls=5,usec=107,usec_per_call=21.40,rejected_calls=0,failed_calls=0
cmdstat_del:calls=46480,usec=25185,usec_per_call=0.54,rejected_calls=0,failed_calls=0
cmdstat_exec:calls=38024729,usec=1166351489,usec_per_call=30.67,rejected_calls=0,failed_calls=0
cmdstat_get:calls=16,usec=5,usec_per_call=0.31,rejected_calls=0,failed_calls=0
# Errorstats
# Latencystats
latency_percentiles_usec_keys:p50=8.031,p99=15.039,p99.9=20.095
latency_percentiles_usec_lindex:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_set:p50=0.001,p99=8.031,p99.9=11.007
latency_percentiles_usec_rpush:p50=0.001,p99=2.007,p99.9=7.007
latency_percentiles_usec_multi:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_flushdb:p50=69.119,p99=69.119,p99.9=69.119
latency_percentiles_usec_config|get:p50=5.023,p99=5.023,p99.9=5.023
latency_percentiles_usec_lrange:p50=2.007,p99=4.015,p99.9=7.007
latency_percentiles_usec_info:p50=18.047,p99=48.127,p99.9=48.127
latency_percentiles_usec_del:p50=0.001,p99=2.007,p99.9=13.055
latency_percentiles_usec_exec:p50=29.055,p99=52.223,p99.9=71.167
latency_percentiles_usec_get:p50=0.001,p99=1.003,p99.9=1.003
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=218,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------
id=19 addr=172.17.0.1:50114 laddr=172.17.0.2:6379 fd=16 name= age=82440 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=210 qbuf=10728 qbuf-free=9746 argv-mem=4 multi-mem=10909 rbs=8192 rbp=5047 obl=2121 oll=0 omem=0 tot-mem=46649 events=r cmd=exec user=default redir=-1 resp=2 lib-name= lib-ver=
id=20 addr=172.17.0.1:34022 laddr=172.17.0.2:6379 fd=17 name= age=82435 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=8192 rbp=5047 obl=0 oll=0 omem=0 tot-mem=29568 events=r cmd=keys user=default redir=-1 resp=2 lib-name= lib-ver=
id=21 addr=172.17.0.1:34032 laddr=172.17.0.2:6379 fd=18 name= age=82435 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=2048 rbp=1024 obl=0 oll=0 omem=0 tot-mem=23424 events=r cmd=exec user=default redir=-1 resp=2 lib-name= lib-ver=
id=22 addr=172.17.0.1:34034 laddr=172.17.0.2:6379 fd=19 name= age=82433 idle=82332 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=exec user=default redir=-1 resp=2 lib-name= lib-ver=
------ CURRENT CLIENT INFO ------
id=19 addr=172.17.0.1:50114 laddr=172.17.0.2:6379 fd=16 name= age=82440 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=210 qbuf=10728 qbuf-free=9746 argv-mem=4 multi-mem=10909 rbs=8192 rbp=5047 obl=2121 oll=0 omem=0 tot-mem=46649 events=r cmd=exec user=default redir=-1 resp=2 lib-name= lib-ver=
argc: '3'
argv[0]: '"LINDEX"'
argv[1]: '"Input Data:S1_V1L_POS"'
argv[2]: '"-1"'
9:M 10 Apr 2024 06:18:32.672 # key 'Input Data:S1_V1L_POS' found in DB containing the following object:
9:M 10 Apr 2024 06:18:32.672 # Object type: 1
9:M 10 Apr 2024 06:18:32.672 # Object encoding: 11
9:M 10 Apr 2024 06:18:32.672 # Object refcount: 1
------ EXECUTING CLIENT INFO ------
id=19 addr=172.17.0.1:50114 laddr=172.17.0.2:6379 fd=16 name= age=82440 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=210 qbuf=10728 qbuf-free=9746 argv-mem=4 multi-mem=10909 rbs=8192 rbp=5047 obl=2121 oll=0 omem=0 tot-mem=46649 events=r cmd=exec user=default redir=-1 resp=2 lib-name= lib-ver=
argc: '3'
argv[0]: '"LINDEX"'
argv[1]: '"Input Data:S1_V1L_POS"'
argv[2]: '"-1"'
9:M 10 Apr 2024 06:18:32.672 # key 'Input Data:S1_V1L_POS' found in DB containing the following object:
9:M 10 Apr 2024 06:18:32.672 # Object type: 1
9:M 10 Apr 2024 06:18:32.672 # Object encoding: 11
9:M 10 Apr 2024 06:18:32.672 # Object refcount: 1
------ MODULES INFO OUTPUT ------
# search_version
search_version:2.8.13
search_redis_version:7.2.4 - oss
# search_index
search_number_of_indexes:0
# search_fields_statistics
# search_dialect_statistics
search_dialect_1:0
search_dialect_2:0
search_dialect_3:0
search_dialect_4:0
# search_runtime_configurations
search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:10000
search_max_aggregate_results:10000
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3
# redisgears_2_trace
redisgears_2_backtrace: 0: redis_module::basic_info_command_handler
1: redisgears::gears_module::__info_func
2: modulesCollectInfo
at /__w/redis-stack/redis-stack/redis/src/module.c:10293:9
3: logModulesInfo
at /__w/redis-stack/redis-stack/redis/src/debug.c:1900:22
printCrashReport
at /__w/redis-stack/redis-stack/redis/src/debug.c:2183:5
4: sigsegvHandler
at /__w/redis-stack/redis-stack/redis/src/debug.c:2164:5
5: <unknown>
6: pthread_kill
7: gsignal
8: abort
9: <unknown>
10: __fortify_fail
11: __stack_chk_fail
12: lindexCommand
at /__w/redis-stack/redis-stack/redis/src/t_list.c:620:1
13: <unknown>
# redisgears_2_UninitialisedBackends
redisgears_2_backend_name:js
# ReJSON_trace
ReJSON_backtrace: 0: redis_module::basic_info_command_handler
1: rejson::__info_func
2: modulesCollectInfo
at /__w/redis-stack/redis-stack/redis/src/module.c:10293:9
3: logModulesInfo
at /__w/redis-stack/redis-stack/redis/src/debug.c:1900:22
printCrashReport
at /__w/redis-stack/redis-stack/redis/src/debug.c:2183:5
4: sigsegvHandler
at /__w/redis-stack/redis-stack/redis/src/debug.c:2164:5
5: <unknown>
6: pthread_kill
7: gsignal
8: abort
9: <unknown>
10: __fortify_fail
11: __stack_chk_fail
12: lindexCommand
at /__w/redis-stack/redis-stack/redis/src/t_list.c:620:1
13: <unknown>
------ CONFIG DEBUG OUTPUT ------
proto-max-bulk-len 512mb
sanitize-dump-payload no
lazyfree-lazy-user-flush no
repl-diskless-sync yes
io-threads 1
io-threads-do-reads no
list-compress-depth 0
lazyfree-lazy-user-del no
replica-read-only yes
lazyfree-lazy-expire no
client-query-buffer-limit 1gb
activedefrag no
repl-diskless-load disabled
slave-read-only yes
lazyfree-lazy-eviction no
lazyfree-lazy-server-del no
------ FAST MEMORY TEST ------
9:M 10 Apr 2024 06:18:32.698 # Bio worker thread #0 terminated
9:M 10 Apr 2024 06:18:32.698 # Bio worker thread #1 terminated
9:M 10 Apr 2024 06:18:32.698 # Bio worker thread #2 terminated
*** Preparing to test memory region 55def2f1a000 (704512 bytes)
*** Preparing to test memory region 55def356f000 (450560 bytes)
*** Preparing to test memory region 7a36ec000000 (135168 bytes)
*** Preparing to test memory region 7a36f0840000 (6291456 bytes)
*** Preparing to test memory region 7a36f2400000 (6291456 bytes)
*** Preparing to test memory region 7a36f2b60000 (5242880 bytes)
*** Preparing to test memory region 7a36f4000000 (135168 bytes)
*** Preparing to test memory region 7a36f85f0000 (4194304 bytes)
*** Preparing to test memory region 7a36f8a00000 (8388608 bytes)
*** Preparing to test memory region 7a36f9200000 (2097152 bytes)
*** Preparing to test memory region 7a36f9600000 (8388608 bytes)
*** Preparing to test memory region 7a36f9e00000 (4194304 bytes)
*** Preparing to test memory region 7a36fa35b000 (8388608 bytes)
*** Preparing to test memory region 7a36fab5c000 (8388608 bytes)
*** Preparing to test memory region 7a36fb35d000 (8388608 bytes)
*** Preparing to test memory region 7a36fbb5e000 (8388608 bytes)
*** Preparing to test memory region 7a36fdda9000 (159744 bytes)
*** Preparing to test memory region 7a36fddd0000 (3670016 bytes)
*** Preparing to test memory region 7a36fe4eb000 (8192 bytes)
*** Preparing to test memory region 7a36fea4d000 (12288 bytes)
*** Preparing to test memory region 7a36fea51000 (8388608 bytes)
*** Preparing to test memory region 7a36ff252000 (8388608 bytes)
*** Preparing to test memory region 7a3700079000 (28672 bytes)
*** Preparing to test memory region 7a3700080000 (9961472 bytes)
*** Preparing to test memory region 7a3700b12000 (4096 bytes)
*** Preparing to test memory region 7a3700b44000 (20480 bytes)
*** Preparing to test memory region 7a3700d65000 (53248 bytes)
*** Preparing to test memory region 7a37011b3000 (12288 bytes)
*** Preparing to test memory region 7a3701343000 (8192 bytes)
*** Preparing to test memory region 7a370137c000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: pthread_kill (base: 0x7a3700bdf8d0)
Module: /lib/x86_64-linux-gnu/libc.so.6 (base 0x7a3700b49000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7a3700bdf8d0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
9:M 10 Apr 2024 06:18:32.842 # dump of function (hexdump of 428 bytes):
f30f1efa41568d56e0415541bd16000000415455534881ec9000000064488b042528000000488984248800000031c083fa01767e4889fb4189f464483b3c25100000000f84c70000004989e641ba0800000031ffb80e0000004c89f2488d352dc113000f0531c0488dab74090000ba01000000f00fb155000f85ca00000080bb730900000074594531ed31d287937409000083fa010f8fbd00000041ba0800000031d24c89f6bf02000000b80e0000000f05488b84248800000064482b0425280000000f859c0000004881c4900000004489e85b5d415c415d415ec30f1f4000448babd0020000e8845605004489e289c74489eeb8ea0000000f053d00f0ffff76854189c541f7ddeb80660f1f440000b8ba0000000f0589c5e8525605004489e289ee89c7b8ea0000000f054189c541f7dd3d00f0ffffb800000000440f46e8e96dffffff0f1f004889efe810a8ffffe929ffffff0f1f004889efe8d0a8ffffe936ffffffe816fb0900660f1f440000f30f1efac3662e0f1f840000000000904157415641554154554889fd534883ec2864488b042528000000488944241831c0648b0425d0020000894424
Function at 0x7a3700c35040 is getpid
Function at 0x7a3700c7f550 is __stack_chk_fail
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/redis/redis/issues
If a Redis module was involved, please open in the module's repo instead.
Suspect RAM error? Use redis-server --test-memory to verify it.
Some other issues could be detected by redis-server --check-system
Aborted (core dumped)
OS: Ubuntu 22.04
PS:
Additionally, upon inspecting the logs preceding the occurrence of a crash, I have observed the recurrence of the message:
9:M 09 Apr 2024 05:05:28.081 * 10000 changes in 60 seconds. Saving...
9:M 09 Apr 2024 05:05:28.081 * Background saving started by pid 288
288:C 09 Apr 2024 05:05:28.084 * DB saved on disk
288:C 09 Apr 2024 05:05:28.084 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
where PID incrementally increases until reaching 288, subsequently resulting in a sigterm signal and what appears to be the automatic restart of the redis-server. For the moment of crash, the PID was 451.
9:signal-handler (1712639140) Received SIGTERM scheduling shutdown...
9:signal-handler (1712639140) Received SIGTERM scheduling shutdown...
9:C 09 Apr 2024 07:19:58.567 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
9:C 09 Apr 2024 07:19:58.567 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
To address the crash issue, I enabled overcommit and configured forced restart in case of crash. But i would like to know the root cause of these crashes for further investigation and resolution.
it's strange that signal SIGTERM is send by the user.
are the signal and the crash at the same time?
@sundb Looking at the time, the SIGTERM was logged 40mins prior to crash. At the log time, no user could have transmitted a shutdown directive to the container. Within the application, the permissible operations are setting, retrieving, and deleting. In the case of application failure, it may disconnect from the container, yet it does not initiate the shutdown process for the container.
@doxg what operation system and version are you using? and how did you install Redis?
i don't think it's SIGTERM sent by a user, it's a fortification assertion. involving LINDEX on a listpack.
if you happen to learn how to reproduce it, please send the reproduction privately to redis@redis.io
@sundb I apologize for late replies. It's Ubuntu 22.04. I installed redis container following tutorial on official website.
https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/docker/
@oranagra I apologize for late replies. I wasn't able to reproduce the issue, but I did stumble upon an intriguing case. Upon investigation, I discovered hardware issues with the current server and requested a replacement. Interestingly, this error hasn't occurred on other machines running different operating systems, such as Ubuntu 18 and Windows 10.
interesting, what type of hardware was it and what was the HW error?
maybe we can blame non-ECC memory for a corruption that led to this.