Facing issue while running Riot redis version 2.4.0

Question

Facing issue while running Riot redis version 2.4.0

abhianand09 opened this issue 4 years ago · comments

Hi,

We have initiated live redis migration from AWS Elasticache to GCP memorystore using Riot-redis tool version 2.4.0(https://github.com/Redislabs-Solution-Architects/riot/releases/download/v2.4.0/riot-redis-2.4.0.zip)
The problem we are facing is this riot process is getting terminated abruptly again and again.
Can you please go through the below logs and let us know if we are missing any setting:

FYI the command which we ran was:
riot-redis -c -h $SOURCE_IP -p 6379 replicate -h $DESTINATION_IP -p 6379 --live

Logs:
18:59:46.804 FINE org.springframework.batch.core.job.AbstractJob : Upgrading JobExecution status: StepExecution: id=1, version=1571, name=LiveRedisClusterKeyDumpItemReader-step, status=FAILED, exitStatus=FAILED, readCount=3181, filterCount=0, writeCount=3173 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1569, rollbackCount=1, exitDescription=java.util.concurrent.TimeoutException

Full logs:

18:59:46.804 FINE org.springframework.batch.core.job.AbstractJob : Upgrading JobExecution status: StepExecution: id=1, version=1571, name=LiveRedisClusterKeyDumpItemReader-step, status=FAILED, exitStatus=FAILED, readCount=3181, filterCount=0, writeCount=3173 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1569, rollbackCount=1, exitDescription=java.util.concurrent.TimeoutException
18:59:46.804 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Committing resourceless transaction on [org.springframework.batch.support.transaction.ResourcelessTransactionManager$ResourcelessTransaction@267bda69]
18:59:46.804 FINE org.springframework.batch.core.job.AbstractJob : Upgrading JobExecution status: StepExecution: id=1, version=1571, name=LiveRedisClusterKeyDumpItemReader-step, status=FAILED, exitStatus=FAILED, readCount=3181, filterCount=0, writeCount=3173 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1569, rollbackCount=1, exitDescription=java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
at org.springframework.batch.item.redis.support.AbstractKeyDumpItemReader.readValues(AbstractKeyDumpItemReader.java:35)
at org.springframework.batch.item.redis.support.AbstractKeyValueItemReader.values(AbstractKeyValueItemReader.java:121)
at org.springframework.batch.item.redis.support.AbstractKeyValueItemReader$ValueReader.write(AbstractKeyValueItemReader.java:107)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:193)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:159)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:294)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:217)
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable.run(TaskExecutorRepeatTemplate.java:262)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:280)
at java.base/java.lang.Thread.run(Thread.java:834)
18:59:46.804 FINE org.springframework.batch.core.job.AbstractJob : Job execution complete: JobExecution: id=0, version=1, startTime=Mon Mar 01 18:55:22 GMT 2021, endTime=null, lastUpdated=Mon Mar 01 18:55:22 GMT 2021, status=FAILED, exitStatus=exitCode=FAILED;exitDescription=java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
at org.springframework.batch.item.redis.support.AbstractKeyDumpItemReader.readValues(AbstractKeyDumpItemReader.java:35)
at org.springframework.batch.item.redis.support.AbstractKeyValueItemReader.values(AbstractKeyValueItemReader.java:121)
at org.springframework.batch.item.redis.support.AbstractKeyValueItemReader$ValueReader.write(AbstractKeyValueItemReader.java:107)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:193)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:159)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:294)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:217)
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable.run(TaskExecutorRepeatTemplate.java:262)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:280)
at java.base/java.lang.Thread.run(Thread.java:834)
, job=[JobInstance: id=0, version=0, Job=[LiveRedisClusterKeyDumpItemReader-job]], jobParameters=[{}]
18:59:46.804 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Creating new transaction with name [org.springframework.batch.core.repository.support.SimpleJobRepository.update]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
18:59:46.807 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Initiating transaction commit
18:59:46.807 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Committing resourceless transaction on [org.springframework.batch.support.transaction.ResourcelessTransactionManager$ResourcelessTransaction@3bbe0782]
18:59:46.807 INFO org.springframework.batch.core.launch.support.SimpleJobLauncher : Job: [SimpleJob: [name=LiveRedisClusterKeyDumpItemReader-job]] completed with the following parameters: [{}] and the following status: [FAILED] in 4m23s910ms
18:59:46.850 FINE org.springframework.batch.repeat.support.RepeatTemplate : Repeat is complete according to policy and result value.
18:59:46.851 FINE org.springframework.batch.core.step.item.ChunkOrientedTasklet : Inputs not busy, ended: true
18:59:46.851 FINE org.springframework.batch.core.step.tasklet.TaskletStep : Applying contribution: [StepContribution: read=0, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
18:59:46.851 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Participating in existing transaction
18:59:46.851 FINE org.springframework.batch.core.step.tasklet.TaskletStep : Saving step execution before commit: StepExecution: id=2, version=2557, name=Listening-step, status=STARTED, exitStatus=EXECUTING, readCount=2556, filterCount=0, writeCount=2556 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=2557, rollbackCount=0, exitDescription=
18:59:46.852 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Participating in existing transaction
18:59:46.852 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Initiating transaction commit
18:59:46.852 FINE org.springframework.batch.support.transaction.ResourcelessTransactionManager : Committing resourceless transaction on [org.springframework.batch.support.transaction.ResourcelessTransactionManager$ResourcelessTransaction@13b059f1]
18:59:46.852 FINE org.springframework.batch.repeat.support.RepeatTemplate : Repeat is complete according to policy and result value.
18:59:46.804 FINE org.springframework.batch.core.job.AbstractJob : Upgrading JobExecution status: StepExecution: id=1, version=1571, name=LiveRedisClusterKeyDumpItemReader-step, status=FAILED, exitStatus=FAILED, readCount=3181, filterCount=0, writeCount=3173 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1569, rollbackCount=1, exitDescription=java.util.concurrent.TimeoutException

Quinton Parker · Answer 1 · Tue Mar 02 2021 18:31:54 GMT+0800 (China Standard Time)

@abhianand09 How many minutes/seconds after initiating the RIOT command does it fail with this error?

Abhishek Anand · Answer 2 · Tue Mar 02 2021 18:34:24 GMT+0800 (China Standard Time)

that is very random @quintonparker .. sometimes it fails after 1 hr sometimes within half an hr sometimes more than 3-4 hrs.
each time it shows similar error with below logs:

Encountered an error executing step LiveRedisClusterKeyDumpItemReader-step in job LiveRedisClusterKeyDumpItemReader-job

Quinton Parker · Answer 3 · Tue Mar 02 2021 18:37:06 GMT+0800 (China Standard Time)

@abhianand09 mind trying the latest release? https://github.com/redis-developer/riot/releases/tag/v2.5.3

There has been some notable bug fixes to live replication on 2.4.x releases

Abhishek Anand · Answer 4 · Tue Mar 02 2021 18:38:12 GMT+0800 (China Standard Time)

sure @quintonparker will update if this error is fixed with latest release

Quinton Parker · Answer 5 · Wed Mar 03 2021 20:40:57 GMT+0800 (China Standard Time)

@abhianand09 hi. any feedback?

Abhishek Anand · Answer 6 · Thu Mar 04 2021 04:00:47 GMT+0800 (China Standard Time)

Hi @quintonparker
Thanks it works fine with latest release and also riot v2.5.3 is copying data across 2 redis clusters very fast now as compared to earlier versions.

Julien Ruaux · Answer 7 · Fri Mar 05 2021 08:34:44 GMT+0800 (China Standard Time)

Thank you for confirming this is working fine. Please reach out if you have any other issue or question.