Reconnection problem in live migration
oguzhanaygn opened this issue · comments
General Information
I've a AWS Redis Elasticache instance that has 174145 keys. Also these keys are sorted set so each one might consists up to 28 million records. I'm trying to do live migration one AWS Elasticache instance to another AWS Elasticache instance (new one). Their versions are the same; both 5.0.6 ,
My riot version: riot-redis v2.15.4
Execution
I used the command below;
./riot-redis --info --uri redis://<elasticache-source>:6379 replicate --type dump --uri redis://<elasticache-target>:6379 --mode live --scan-count 50 --reader-batch 10 --reader-queue 10 --reader-pool 1 --threads 3
This command has these lines in standard output;
Scanning 26% │███████████████████████████████████▉ │ 45300/174145 (0:00:52 / 0:02:27) 871.2/s
Reconnecting, last destination was <elasticache-target>:6379
Scanning 46% │███████████████████████████████████████████████████████████████▉ │ 80700/174145 (0:01:24 / 0:01:38) 960.7/s
Reconnecting, last destination was <elasticache-target>:6379
Scanning 97% │█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ │ 170200/174145 (0:01:39 / 0:00:02) 1719.2/s
Step: [redis-scan-reader] executed in 1m39s428ms
Closing connection pool
Scanning 100% │█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 174145/174145 (0:02:24 / 0:00:00) 1209.3/s
Scanning 100% │█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 174145/174145 (0:02:24 / 0:00:00) 1209.3/s
Step: [scan-replication-step] executed in 2m25s147ms
Closing redis-scan-reader
Closing connection pool
Problem
Eventhough it seems finished, When I ran riot with compare subcommand, it gives,
Verifying 100% │████████████████████████████████████████████████████████████████████████████████████████████████████████│ 174145/174145 (0:02:01 / 0:00:00) 1439.2/s >92 T0 ≠0 ⧗05 (0:00:00 / ?) ?/s
Verifying 100% │████████████████████████████████████████████████████████████████████████████████████████████████████████│ 174145/174145 (0:02:01 / 0:00:00) 1439.2/s >92 T0 ≠0 ⧗0
There are 92 keys which exist in source but don't exist in data.
Funny thing is, I stopped the replication and started again, when it's finished, I ran compare again, difference decreased to 54,
For the 3rd turn, difference was only 17 keys.
I didn't understand the problem clearly, when Reconnecting, last destination was ... shows up, one of the CPU cores becomes %100 at the same time. Even if I'm running the command with multi threading, only 1 CPU core becomes %100 and others becomes idle.
Questions
1- Is this situation happening because of one of the huge keys? Because there are few of them and this Reconnecting, last destination was ... message shows up few times.
2- What can I do to fix this situation, is it possible to use riot in these circumstances? Is it possible to distribute CPU consumption to compensate reading/writing huge keys?
3- Can there be any other reason for this situation?
4- How can I understand which key is causing the problem?
Thank you very much.
This looks like an issue with live replication on big keys as explained in the documentation: https://developer.redis.com/riot/riot-redis/index.html#live-replication
Unfortunately there is not much RIOT can do in these situations as every update to a collection key means replicating the whole key over again...