Facing issue while running Riot redis version 2.4.0
abhianand09 opened this issue · comments
Hi,
We have initiated live redis migration from AWS Elasticache to GCP memorystore using Riot-redis tool version 2.4.0(https://github.com/Redislabs-Solution-Architects/riot/releases/download/v2.4.0/riot-redis-2.4.0.zip)
The problem we are facing is this riot process is getting terminated abruptly again and again.
Can you please go through the below logs and let us know if we are missing any setting:
FYI the command which we ran was:
riot-redis -c -h $SOURCE_IP -p 6379 replicate -h $DESTINATION_IP -p 6379 --live
Logs:
18:59:46.804 FINE org.springframework.batch.core.job.AbstractJob : Upgrading JobExecution status: StepExecution: id=1, version=1571, name=LiveRedisClusterKeyDumpItemReader-step, status=FAILED, exitStatus=FAILED, readCount=3181, filterCount=0, writeCount=3173 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1569, rollbackCount=1, exitDescription=java.util.concurrent.TimeoutException
Full logs:
18:59:46.804 FINE org.springframework.batch.core.job.AbstractJob : Upgrading JobExecution status: StepExecution: id=1, version=1571, name=LiveRedisClusterKeyDumpItemReader-step, status=FAILED, exitStatus=FAILED, readCount=3181, filterCount=0, writeCount=3173 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1569, rollbackCount=1, exitDescription=java.util.concurrent.TimeoutException
18:59:46.804 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Committing resourceless transaction on [org.springframework.batch.support.transaction.ResourcelessTransactionManager$ResourcelessTransaction@267bda69]
18:59:46.804 FINE org.springframework.batch.core.job.AbstractJob : Upgrading JobExecution status: StepExecution: id=1, version=1571, name=LiveRedisClusterKeyDumpItemReader-step, status=FAILED, exitStatus=FAILED, readCount=3181, filterCount=0, writeCount=3173 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1569, rollbackCount=1, exitDescription=java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
at org.springframework.batch.item.redis.support.AbstractKeyDumpItemReader.readValues(AbstractKeyDumpItemReader.java:35)
at org.springframework.batch.item.redis.support.AbstractKeyValueItemReader.values(AbstractKeyValueItemReader.java:121)
at org.springframework.batch.item.redis.support.AbstractKeyValueItemReader$ValueReader.write(AbstractKeyValueItemReader.java:107)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:193)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:159)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:294)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:217)
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable.run(TaskExecutorRepeatTemplate.java:262)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:280)
at java.base/java.lang.Thread.run(Thread.java:834)
18:59:46.804 FINE org.springframework.batch.core.job.AbstractJob : Job execution complete: JobExecution: id=0, version=1, startTime=Mon Mar 01 18:55:22 GMT 2021, endTime=null, lastUpdated=Mon Mar 01 18:55:22 GMT 2021, status=FAILED, exitStatus=exitCode=FAILED;exitDescription=java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
at org.springframework.batch.item.redis.support.AbstractKeyDumpItemReader.readValues(AbstractKeyDumpItemReader.java:35)
at org.springframework.batch.item.redis.support.AbstractKeyValueItemReader.values(AbstractKeyValueItemReader.java:121)
at org.springframework.batch.item.redis.support.AbstractKeyValueItemReader$ValueReader.write(AbstractKeyValueItemReader.java:107)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:193)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:159)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:294)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:217)
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable.run(TaskExecutorRepeatTemplate.java:262)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:280)
at java.base/java.lang.Thread.run(Thread.java:834)
, job=[JobInstance: id=0, version=0, Job=[LiveRedisClusterKeyDumpItemReader-job]], jobParameters=[{}]
18:59:46.804 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Creating new transaction with name [org.springframework.batch.core.repository.support.SimpleJobRepository.update]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
18:59:46.807 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Initiating transaction commit
18:59:46.807 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Committing resourceless transaction on [org.springframework.batch.support.transaction.ResourcelessTransactionManager$ResourcelessTransaction@3bbe0782]
18:59:46.807 INFO org.springframework.batch.core.launch.support.SimpleJobLauncher : Job: [SimpleJob: [name=LiveRedisClusterKeyDumpItemReader-job]] completed with the following parameters: [{}] and the following status: [FAILED] in 4m23s910ms
18:59:46.850 FINE org.springframework.batch.repeat.support.RepeatTemplate : Repeat is complete according to policy and result value.
18:59:46.851 FINE org.springframework.batch.core.step.item.ChunkOrientedTasklet : Inputs not busy, ended: true
18:59:46.851 FINE org.springframework.batch.core.step.tasklet.TaskletStep : Applying contribution: [StepContribution: read=0, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
18:59:46.851 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Participating in existing transaction
18:59:46.851 FINE org.springframework.batch.core.step.tasklet.TaskletStep : Saving step execution before commit: StepExecution: id=2, version=2557, name=Listening-step, status=STARTED, exitStatus=EXECUTING, readCount=2556, filterCount=0, writeCount=2556 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=2557, rollbackCount=0, exitDescription=
18:59:46.852 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Participating in existing transaction
18:59:46.852 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Initiating transaction commit
18:59:46.852 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Committing resourceless transaction on [org.springframework.batch.support.transaction.ResourcelessTransactionManager$ResourcelessTransaction@13b059f1]
18:59:46.852 FINE org.springframework.batch.repeat.support.RepeatTemplate : Repeat is complete according to policy and result value.
18:59:46.804 FINE org.springframework.batch.core.job.AbstractJob : Upgrading JobExecution status: StepExecution: id=1, version=1571, name=LiveRedisClusterKeyDumpItemReader-step, status=FAILED, exitStatus=FAILED, readCount=3181, filterCount=0, writeCount=3173 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1569, rollbackCount=1, exitDescription=java.util.concurrent.TimeoutException
@abhianand09 How many minutes/seconds after initiating the RIOT command does it fail with this error?
that is very random @quintonparker .. sometimes it fails after 1 hr sometimes within half an hr sometimes more than 3-4 hrs.
each time it shows similar error with below logs:
Encountered an error executing step LiveRedisClusterKeyDumpItemReader-step in job LiveRedisClusterKeyDumpItemReader-job
@abhianand09 mind trying the latest release? https://github.com/redis-developer/riot/releases/tag/v2.5.3
There has been some notable bug fixes to live replication on 2.4.x releases
sure @quintonparker will update if this error is fixed with latest release
@abhianand09 hi. any feedback?
Hi @quintonparker
Thanks it works fine with latest release and also riot v2.5.3 is copying data across 2 redis clusters very fast now as compared to earlier versions.
Thank you for confirming this is working fine. Please reach out if you have any other issue or question.