1G的dump.rdb迁移过程中16G内存跑满，然后被kill

[rabbit-yu]

问题描述（Issue Description）

1G的dump.rdb迁移过程中16G内存跑满，然后被kill

环境信息（Environment）

配置：
[rdb_reader]
filepath = "/root/redis-s/dump.rdb"

[redis_writer]
cluster = false # set to true if target is a redis cluster
address = "127.0.0.1:6379" # when cluster is true, set address to one of the cluster node
username = "" # keep empty if not using ACL
password = "" # keep empty if no authentication is required
tls = false

[advanced]
dir = "data"
ncpu = 0 # runtime.GOMAXPROCS, 0 means use runtime.NumCPU() cpu cores
pprof_port = 0 # pprof port, 0 means disable
status_port = 0 # status port, 0 means disable

log_file = "shake.log"
log_level = "info" # debug, info or warn
log_interval = 5 # in seconds

rdb_restore_command_behavior = "panic" # panic, rewrite or skip

pipeline_count_limit = 1024

target_redis_client_max_querybuf_len = 1024_000_000

target_redis_proto_max_bulk_len = 512_000_000

aws_psync = "" # example: aws_psync = "10.0.0.1:6379@nmfu2sl5osync,10.0.0.1:6379@xhma21xfkssync"

[module]
target_mbbloom_version = 20603

日志信息（Logs）

2024-01-10 02:40:37 INF load config from file: shake.toml
2024-01-10 02:40:37 INF log_level: [info], log_file: [/root/redis-s/data/shake.log]
2024-01-10 02:40:37 INF changed work dir to [/root/redis-s/data]
2024-01-10 02:40:37 INF GOMAXPROCS defaults to the value of runtime.NumCPU [8]
2024-01-10 02:40:37 INF not set pprof port
2024-01-10 02:40:37 INF no function script
2024-01-10 02:40:37 INF create RdbReader: /root/redis-s/dump.rdb
2024-01-10 02:40:37 INF create RedisStandaloneWriter: 127.0.0.1:6379
2024-01-10 02:40:37 INF not set status port
2024-01-10 02:40:37 INF start syncing...
2024-01-10 02:40:37 INF [rdb_reader] start read
2024-01-10 02:40:42 INF read_count=[2], read_ops=[0.00], write_count=[2], write_ops=[0.00], init
2024-01-10 02:40:47 INF read_count=[2], read_ops=[0.00], write_count=[2], write_ops=[0.00], init
2024-01-10 02:40:52 INF read_count=[2], read_ops=[0.00], write_count=[2], write_ops=[0.00], init
2024-01-10 02:40:57 INF read_count=[2], read_ops=[0.00], write_count=[2], write_ops=[0.00], init
2024-01-10 02:41:03 INF read_count=[2], read_ops=[0.00], write_count=[2], write_ops=[0.00], init
Killed

[suxb201]

1. pipeline_count_limit 改到 1 试试
2. 可能有大 key，导致 shake 在解析时占用大量内存。可以分析下源端的 Redis 内存占用，看看有没有个别 key 占用了大量内存。

[rabbit-yu]

1. pipeline_count_limit 改到 1 试试
2. 可能有大 key，导致 shake 在解析时占用大量内存。可以分析下源端的 Redis 内存占用，看看有没有个别 key 占用了大量内存。

确实有大key，但是改成1也是内存跑满，被杀死

[suxb201]

@rabbit-yu 大 key 无解

[rabbit-yu]

@rabbit-yu 大 key 无解

好吧