CALL db.idx.vector.queryRelationships crashes graph server if vector length used differs from that stored in the index.
schwab opened this issue · comments
When a vector index defined with a specific size (in this case the vector is defined with 384 values) but a query is passed which has a different number of values than it was defined with, it causes the server to crash.
The db.indexes call looks like this (but I noticed it does not show the number of dimensions expected in info).
1) 1) "label"
2) "properties"
3) "types"
4) "language"
5) "stopwords"
6) "entitytype"
7) "status"
8) "info"
2) 1) 1) "is_a"
2) "[vector]"
3) "{vector: [VECTOR]}"
4) "english"
5) "[]"
6) "RELATIONSHIP"
7) "OPERATIONAL"
8) "{gcPolicy: 0, score: 1.000000, lang: english, fields: [{path: vector:vector, name: vector:vector, options: 0, textWeight: 0.000000, tagCaseSensitive: false}, {path: NONE_INDEXABLE_FIELDS, name: NONE_INDEXABLE_FIELDS, options: 0, textWeight: 0.000000, tagCaseSensitive: true}, {path: range:_src_id, name: range:_src_id, options: 0, textWeight: 0.000000, tagCaseSensitive: false}, {path: range:_dest_id, name: range:_dest_id, options: 0, textWeight: 0.000000, tagCaseSensitive: false}], numDocuments: 1387, maxDocId: 1387, docTableSize: 124830, sortablesSize: 0, docTrieSize: 47676, numTerms: 0, numRecords: 4161, invertedSize: 11065, invertedCap: 0, skipIndexesSize: 0, scoreIndexesSize: 0, offsetVecsSize: 0, offsetVecRecords: 0, termsSize: 0, indexingFailures: 0, totalCollected: 0, numCycles: 0, totalMSRun: 0, lastRunTimeMs: 0}"
🛑 This call crashes the server (it has substantially less values than the expected 384 value vector).
CALL db.idx.vector.queryRelationships('is_a','vector',10, vecf32([-0.087367, 0.069268, -0.041078, 0.015115, -0.010957, 0.021993, 0.111647, 0.032519, 0.077085, 0.045142, 0.00692, -0.035153, -0.065759, -0.018555, -0.048009, 0.051776, -0.010507])) YIELD relationship, score RETURN relationship, score as Score
The server does respond with this error message, but it doesn't tell us much about why it failed.
Reached the max retries per request limit (which is 1). Refer to \"maxRetriesPerRequest\" option for details.
🎯 When a properly sized query is passed though the server does not fail and the query succeeds.
Here's the server trace...
=== REDIS BUG REPORT START: Cut & paste starting from here ===
8:M 07 Dec 2023 20:25:58.640 # Redis 7.2.3 crashed by signal: 11, si_code: 1
8:M 07 Dec 2023 20:25:58.640 # Accessing address: (nil)
8:M 07 Dec 2023 20:25:58.640 # Crashed running the instruction at: 0x7fba6a9bed0f
------ STACK TRACE ------
EIP:
/FalkorDB/bin/linux-x64-release/src/falkordb.so(RediSearch_ResultsIteratorNext+0x1f)[0x7fba6a9bed0f]
Backtrace:
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fba6bc03520]
/FalkorDB/bin/linux-x64-release/src/falkordb.so(RediSearch_ResultsIteratorNext+0x1f)[0x7fba6a9bed0f]
/FalkorDB/bin/linux-x64-release/src/falkordb.so(+0x295f38)[0x7fba6870ff38]
/FalkorDB/bin/linux-x64-release/src/falkordb.so(Proc_Step+0x13)[0x7fba68710b53]
/FalkorDB/bin/linux-x64-release/src/falkordb.so(+0x270e14)[0x7fba686eae14]
/FalkorDB/bin/linux-x64-release/src/falkordb.so(+0x2717dc)[0x7fba686eb7dc]
/FalkorDB/bin/linux-x64-release/src/falkordb.so(+0x271c44)[0x7fba686ebc44]
/FalkorDB/bin/linux-x64-release/src/falkordb.so(ExecutionPlan_Execute+0x50)[0x7fba686da460]
/FalkorDB/bin/linux-x64-release/src/falkordb.so(+0x253dc5)[0x7fba686cddc5]
/FalkorDB/bin/linux-x64-release/src/falkordb.so(+0x2b156f)[0x7fba6872b56f]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7fba6bc55ac3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7fba6bce6bf4]
------ REGISTERS ------
8:M 07 Dec 2023 20:25:58.642 #
RAX:0000000000000000 RBX:0000000000000000
RCX:00007fb9bd481e00 RDX:00007fba615f6070
RDI:0000000000000000 RSI:00007fb9bdf17fc0
RBP:00007fb9bdf180a0 RSP:00007fba615f6040
R8 :0000000000000000 R9 :00007fb9bd481e00
R10:00007fba615f6d60 R11:00007fba615f6dc0
R12:00007fba615f6070 R13:0000000000000008
R14:00007fb9bd400600 R15:00007fb9be0af2c0
RIP:00007fba6a9bed0f EFL:0000000000010202
CSGSFS:002b000000000033
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f604f) -> 00007fb9be000840
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f604e) -> 00007fb9c03b06f0
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f604d) -> 00007fba686eae14
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f604c) -> 00007fba341712c0
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f604b) -> 00007fba68710b53
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f604a) -> 0000000000000004
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f6049) -> 00007fb9be02dae0
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f6048) -> 00007fb9bd420000
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f6047) -> ef56ab7e9b65a500
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f6046) -> 0000000000000000
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f6045) -> 00007fba6870ff38
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f6044) -> 00007fb9be02daec
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f6043) -> 0000000000000004
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f6042) -> 00007fb9be02dae0
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f6041) -> 00007fb9bd400780
8:M 07 Dec 2023 20:25:58.642 # (00007fba615f6040) -> 00007fb9bd400780
------ INFO OUTPUT ------
# Server
redis_version:7.2.3
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7f52fd1717e1b756
redis_mode:standalone
os:Linux 6.5.6-76060506-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.4.0
process_id:8
process_supervised:no
run_id:ad4cf6fd1437b2a06d8438f4fbade1b609066362
tcp_port:6379
server_time_usec:1701980758634099
uptime_in_seconds:76254
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7481942
executable:/FalkorDB/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=*,bind=-::*,port=6379
# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:0
blocked_clients:1
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
# Memory
used_memory:3508216968
used_memory_human:3.27G
used_memory_rss:1668677632
used_memory_rss_human:1.55G
used_memory_peak:3508284584
used_memory_peak_human:3.27G
used_memory_peak_perc:100.00%
used_memory_overhead:959512
used_memory_startup:954648
used_memory_dataset:3507257456
used_memory_dataset_perc:100.00%
allocator_allocated:3512867576
allocator_active:3542171648
allocator_resident:3580227584
total_system_memory:67269480448
total_system_memory_human:62.65G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.01
allocator_frag_bytes:29304072
allocator_rss_ratio:1.01
allocator_rss_bytes:38055936
rss_overhead_ratio:0.47
rss_overhead_bytes:-1911549952
mem_fragmentation_ratio:0.48
mem_fragmentation_bytes:-1839124904
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:3856
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:1286
rdb_bgsave_in_progress:0
rdb_last_save_time:1701980612
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:12
rdb_current_bgsave_time_sec:-1
rdb_saves:10
rdb_last_cow_size:40513536
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:44
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:18005
total_commands_processed:146779
instantaneous_ops_per_sec:0
total_net_input_bytes:89371238
total_net_output_bytes:115415835
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:1136
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:108016
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:99479
total_forks:11
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:24
dump_payload_sanitizations:0
total_reads_processed:164789
total_writes_processed:147692
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:2709
reply_buffer_expands:2627
eventloop_cycles:1046446
eventloop_duration_sum:703309227
eventloop_duration_cmd_sum:2178978
instantaneous_eventloop_cycles_per_sec:9
instantaneous_eventloop_duration_usec:932
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:6b20f433a7368d6e86659a064e52f505a332f1f3
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:105.651102
used_cpu_user:792.432229
used_cpu_sys_children:23.996639
used_cpu_user_children:93.748801
used_cpu_sys_main_thread:0.201578
used_cpu_user_main_thread:4.567770
# Modules
module:name=graph,ver=999999,api=1,filters=0,usedby=[],using=[],options=[]
# Commandstats
cmdstat_info:calls=2627,usec=386062,usec_per_call=146.96,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=17998,usec=82410,usec_per_call=4.58,rejected_calls=0,failed_calls=0
cmdstat_graph.RO_QUERY:calls=146,usec=16352232,usec_per_call=112001.59,rejected_calls=0,failed_calls=15
cmdstat_client|setname:calls=7,usec=21,usec_per_call=3.00,rejected_calls=0,failed_calls=0
cmdstat_client|list:calls=1,usec=43,usec_per_call=43.00,rejected_calls=0,failed_calls=0
cmdstat_client|setinfo:calls=35996,usec=31385,usec_per_call=0.87,rejected_calls=0,failed_calls=0
cmdstat_module|list:calls=1,usec=14,usec_per_call=14.00,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=3,usec=72,usec_per_call=24.00,rejected_calls=0,failed_calls=0
cmdstat_graph.QUERY:calls=90000,usec=130413031,usec_per_call=1449.03,rejected_calls=0,failed_calls=9
# Errorstats
errorstat_'u':count=1
errorstat_Query:count=15
errorstat_Unknown:count=4
errorstat_errMsg_:count=4
# Latencystats
latency_percentiles_usec_info:p50=142.335,p99=299.007,p99.9=516.095
latency_percentiles_usec_auth:p50=4.015,p99=13.055,p99.9=30.079
latency_percentiles_usec_graph.RO_QUERY:p50=301.055,p99=1002438.655,p99.9=1002438.655
latency_percentiles_usec_client|setname:p50=3.007,p99=5.023,p99.9=5.023
latency_percentiles_usec_client|list:p50=43.007,p99=43.007,p99.9=43.007
latency_percentiles_usec_client|setinfo:p50=1.003,p99=3.007,p99.9=8.031
latency_percentiles_usec_module|list:p50=14.015,p99=14.015,p99.9=14.015
latency_percentiles_usec_config|get:p50=14.015,p99=52.223,p99.9=52.223
latency_percentiles_usec_graph.QUERY:p50=232.447,p99=9568.255,p99.9=11862.015
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=7,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------
id=48891 addr=172.17.0.1:53996 laddr=172.17.0.3:6379 fd=12 name=redisinsight-common-ca4f8320 age=13084 idle=3 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=info user=default redir=-1 resp=2 lib-name= lib-ver=
id=59135 addr=172.17.0.1:49784 laddr=172.17.0.3:6379 fd=9 name=redisinsight-workbench-ca4f8320 age=185 idle=0 flags=b db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=3542 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=25966 events=r cmd=graph.QUERY user=default redir=-1 resp=2 lib-name= lib-ver=
------ MODULES INFO OUTPUT ------
# graph_executing commands
graph_command:graph.query CALL db.idx.vector.queryRelationships('is_a','vector',10, vecf32([-0.087367, 0.069268, -0.041078, 0.015115, -0.010957, 0.021993, 0.111647, 0.032519, 0.077085, 0.045142, 0.00692, -0.035153, -0.065759, -0.018555, -0.048009, 0.051776, -0.010507, 9e-05, -0.07451, -0.082257, 0.009024, -0.001587, -0.009134, 0.054776, -0.044789, 0.049455, 0.006802, 0.027898, 0.105398, -0.08699, 0.033794, -0.081994, 0.059517, 0.040457, -0.036408, -0.025839, 0.036618, -0.003736, -0.046661, 0.010938, 0.036768, -0.027147, -0.091995, -0.055288, -0.025965, -0.091414, -0.050829, -0.028697, 0.028014, 0.0106, 0.020364, -0.002523, -0.04075, -0.000155, -0.012151, 0.023521, 0.027248, -0.032646, 0.018607, 0.014233, 0.064329, -0.010854, -0.059181, 0.055017, -0.074586, -0.016054, -0.008359, 0.014309, -0.095094, -0.024111, 0.033689, -0.041449, -0.015169, -0.02006, 0.022172, 0.03366, 0.017967, -0.059373, 0.003079, -0.034155, 0.070184, -0.087854, -0.043151, -0.043858, -0.046516, 0.000299, 0.031314, -0.01003, 0.044885, -0.02596, -0.08297, -0.032372, 0.065977, 0.026276, -0.033084, 0.027759, 0.029365, -0.004315, -0.087797, 0.112924, -0.049979, 0.139509, -0.030387, -0.04157, -0.086048, 0.020967, 0.010252, 0.066231, 0.002105, -0.008525, -0.06965, 0.005499, -0.070791, 0.117376, -0.008108, 0.006258, -0.075912, -0.008417, -0.003266, -0.057811, -0.007951, -0.038788, -0.079824, 0.057485, -0.046375, -0.002165, 0.010849, -0.0, 0.046644, -0.006194, 0.012439, -0.003731, 0.005866, 0.03006, -0.038544, -0.018496, -0.06631, 0.000172, 0.001948, -0.000639, -0.036718, 0.006248, 0.171928, 0.108895, -0.055306, 0.00909, 0.005408, -0.038537, 0.02252, 0.072178, -0.038806, 0.038877, 0.067923, 0.031742, -0.049922, -0.05422, 0.025744, 0.060456, 0.000385, 0.055181, -0.059828, 0.03809, -0.070292, 0.035179, -0.027837, -0.018831, -0.061265, -0.009789, 0.046185, -0.020514, 0.02947, 0.004862, -0.03627, 0.034125, 0.052884, 0.041968, 0.094806, 0.047706, -0.084829, -0.014786, -0.046331, -0.001092, 0.040731, -0.04781, -0.043186, 0.06204, -0.000684, -0.011604, 0.136117, 0.037929, -0.035224, 0.09798, -0.061461, -0.075437, 0.036533, -0.004713, 0.07479, -0.039154, -0.100407, 0.013149, 0.014607, 0.050211, -0.000231, -0.133556, 0.043121, 0.044009, -0.031464, 0.030754, -0.062084, 0.052602, -0.037296, 0.004949, 0.043385, 0.068476, -0.011467, -0.050026, -0.079307, 0.041818, -0.102149, 0.063866, 0.021571, 0.083347, -0.065479, 0.0, 0.035159, -0.075568, 0.109221, 0.068139, 0.022454, 0.006801, 0.091387, 0.092222, -0.011731, -0.024079, 0.03553, -0.025494, 0.094489, 0.021887, 0.05167, 0.016625, 0.068508, -0.130901, -0.014483, 0.014332, 0.053195, 0.073881, 0.003056, -0.0355, 0.002523, 0.001954, 0.076264, -0.035428, -0.111267, 0.01098, -0.020275, 0.025382, -0.068558, 0.071159, -0.123679, 0.025228, 0.030494, -0.006341, -0.059331, 0.029591, -0.052126, 0.004757, -0.038366, 0.090591, 0.048835, 0.004909, -0.023082, 0.003137, -0.011243, -0.042723, 0.002975, 0.016887, -0.070554, 0.079775, 0.06739, -0.019166, -0.003421, -0.047763, -0.065324, -0.046058, 0.045364, 0.06816, -5e-06, 0.009029, -0.035823, -0.02077, 0.054437, -0.036172, -0.013881, -0.027654, 0.056956, -0.010237, -0.079498, 0.081476, -0.086671, 0.02568, -0.042459, 0.011759, 0.00411, -0.006491, -0.069219, 0.008921, -0.067795, 0.072649, -0.024587, 0.066138, 0.012509, -0.021047, 0.046765, 0.052081, 0.006829, 0.039428, 0.056016, -0.036282, -0.014306, -0.0, 0.002545, 0.010398, -0.017746, -0.0081, 0.026874, -0.006626, -0.077725, -0.063751])) YIELD relationship, score RETURN relationship, score as Score
------ CONFIG DEBUG OUTPUT ------
repl-diskless-sync yes
proto-max-bulk-len 512mb
list-compress-depth 0
lazyfree-lazy-server-del no
sanitize-dump-payload no
activedefrag no
io-threads 1
lazyfree-lazy-eviction no
io-threads-do-reads no
lazyfree-lazy-user-del no
lazyfree-lazy-expire no
repl-diskless-load disabled
lazyfree-lazy-user-flush no
replica-read-only yes
slave-read-only yes
client-query-buffer-limit 1gb
------ FAST MEMORY TEST ------
8:M 07 Dec 2023 20:25:58.644 # main thread terminated
8:M 07 Dec 2023 20:25:58.644 # Bio worker thread #0 terminated
8:M 07 Dec 2023 20:25:58.645 # Bio worker thread #1 terminated
8:M 07 Dec 2023 20:25:58.645 # Bio worker thread #2 terminated
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: RediSearch_ResultsIteratorNext (base: 0x7fba6a9becf0)
Module: /FalkorDB/bin/linux-x64-release/src/falkordb.so (base 0x7fba6847a000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7fba6a9becf0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
8:M 07 Dec 2023 20:25:58.645 # dump of function (hexdump of 159 bytes):
f30f1efa41554c8d6f0841544989d455488daee0000000534889fb4883ec08488b034c89ee488b7808ff504085c00f84ac000000488b43084889ef488b30e87df706004885c074d7f6401b0175d1488943104d85e47438488b78080fb64fff89ca83e20780fa040f87d80dcefd488d35e8f613000fb6d2486314964801f23effe20f1f80000000008b4ff749890c24488b40084883c4085b5d415c415dc366
Function at 0x7fba6aa2e4b0 is DocTable_Get
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/redis/redis/issues
If a Redis module was involved, please open in the module's repo instead.
Suspect RAM error? Use redis-server --test-memory to verify it.
Some other issues could be detect
Thank you @schwab,
Apologize for the inconvenience, I'll look into it ASAP.