Error when replicating data between Elasticache for Redis 6 to 7
giulianisanches opened this issue · comments
Hi everyone,
I don't know if this is supported, but I'm trying to migrate some data from my AWS Elasticache running with Redis 6.2.6 to a new one with Redis 7.0.
Using the option --dry-run works just fine, but when i remove it to transfer the data i get the following error:
EDITED: Added --stacktrace (what is that skip limit?)
[user@host ~]$ riot-redis --stacktrace -c -h "$ORIGIN_REDIS_ENDPOINT" -p "$REDIS_PORT" -a "$REDIS_PASSWORD" --tls replicate -c -h "$DESTINATION_REDIS_ENDPOINT" -p "$REDIS_PORT" -a "$REDIS_PASSWORD" --tls --scan-match='l_*' --reader-batch 50 --batch 25
Scanning 0% ╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/3337 (0:00:00 / ?)
12:58:52.237 SEVERE org.springframework.batch.core.step.AbstractStep : Encountered an error executing step snapshot-replication in job snapshot-replication
org.springframework.batch.core.step.skip.SkipLimitExceededException: Skip limit of '0' exceeded
at org.springframework.batch.core.step.skip.LimitCheckingItemSkipPolicy.shouldSkip(LimitCheckingItemSkipPolicy.java:133)
at org.springframework.batch.core.step.skip.CompositeSkipPolicy.shouldSkip(CompositeSkipPolicy.java:41)
at org.springframework.batch.core.step.skip.ExceptionClassifierSkipPolicy.shouldSkip(ExceptionClassifierSkipPolicy.java:70)
at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.shouldSkip(FaultTolerantChunkProcessor.java:519)
at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.checkSkipPolicy(FaultTolerantChunkProcessor.java:550)
at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.scan(FaultTolerantChunkProcessor.java:632)
at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.access$900(FaultTolerantChunkProcessor.java:56)
at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor$5.recover(FaultTolerantChunkProcessor.java:434)
at org.springframework.retry.support.RetryTemplate.handleRetryExhausted(RetryTemplate.java:539)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:387)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:255)
at org.springframework.batch.core.step.item.BatchRetryTemplate.execute(BatchRetryTemplate.java:217)
at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.write(FaultTolerantChunkProcessor.java:444)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:217)
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:152)
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:413)
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:136)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:149)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140)
at com.redis.spring.batch.common.JobRunner.run(JobRunner.java:77)
at com.redis.riot.AbstractJobCommand.call(AbstractJobCommand.java:37)
at com.redis.riot.AbstractJobCommand.call(AbstractJobCommand.java:19)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
at com.redis.riot.RiotApp.execute(RiotApp.java:57)
at com.redis.riot.RiotApp.lambda$commandLine$1(RiotApp.java:70)
at picocli.CommandLine.execute(CommandLine.java:2170)
at com.redis.riot.RiotApp.execute(RiotApp.java:65)
at com.redis.riot.redis.RiotRedis.main(RiotRedis.java:12)
Caused by: io.lettuce.core.RedisCommandExecutionException: ERR DUMP payload version or checksum are wrong
at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:147)
at io.lettuce.core.internal.Exceptions.bubble(Exceptions.java:72)
at io.lettuce.core.internal.Exceptions.fromSynchronization(Exceptions.java:102)
at io.lettuce.core.internal.Futures.awaitAll(Futures.java:226)
at io.lettuce.core.LettuceFutures.awaitAll(LettuceFutures.java:59)
at com.redis.spring.batch.RedisItemWriter.write(RedisItemWriter.java:44)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:193)
at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.scan(FaultTolerantChunkProcessor.java:609)
... 41 more
Caused by: io.lettuce.core.RedisCommandExecutionException: ERR DUMP payload version or checksum are wrong
at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:147)
at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:116)
at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:120)
at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:111)
at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:63)
at io.lettuce.core.cluster.ClusterCommand.complete(ClusterCommand.java:65)
at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:63)
at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:747)
at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:682)
at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:599)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
I have added the options --skip-policy=LIMIT --skip-limit=5 but the error still says Skip limit of '0' exceeded
the command riot-redis replicate --help show the following:
Reader options
--reader-queue=<int> Capacity of the reader queue (default: 10000).
--reader-threads=<int> Number of reader threads (default: 1).
--reader-batch=<int> Number of reader values to process at once
(default: 50).
--reader-skip-policy=<name>
Policy to determine if some reading should be
skipped: ALWAYS, NEVER, LIMIT (default: NEVER).
--reader-skip-limit=<int>
LIMIT skip policy: max number of failed items
before considering reader has failed (default:
0).
--scan-match=<glob> SCAN MATCH pattern (default: *).
--scan-count=<int> SCAN COUNT option (default: 1000).
--scan-type=<type> SCAN TYPE option.I have tried to chante both --reader-skip-policy and --reader-skip-count but the error persists.
I kind of have resolved the issue using replicate-ds instead of replicate. But now the scan count is differente than the number of items being replicated, but thats another history.