Persistence failure when replaying events

[Hassaan Ahmed]

We have an Akka.Cluster deployment where one of our Nodes is hosted as a Topshelf service. The Node has single instance and uses Akka.Persistence.Redis for Actor state recovery in case of Service restart/upgrades. Deployment script successfully performs these steps: stop->uninstall->reinstall new version->start the service. We can see that Journals are present with correct state. However, after the service is started, we see this error in NLog causing state loss.

Persistence failure when replaying events for persistenceId [workerSupervisorActor]. Last known sequence number [0]

In Redis, we can see "PersistenceId": "workerSupervisorActor", "SequenceNr": 1
However in error description it says Last known sequence number [0], which means call to RedisJournal.ReadHighestSequenceNrAsync() is failing.

Connection String to Azure Redis Cache looks like this
xxx.redis.cache.windows.net:6380,password=xxx=,ssl=True,abortConnect=False,connectTimeout=10000,connectRetry=3

We're using these Nuget Packages:
Akka.Persistence: 1.1.1.28-beta
Akka.Persistence.Redis: 0.1.0-beta
StackExchange.Redis: 1.1.603

[Bartosz Sypytkowski]

This was a part of the talk on gitter: the most probable point of failure is a fail thrown when reading ReadHighestSequenceNr (error msg shows seqNr = 0 at the point of failure). Since this is a first call to Redis, the problem may be caused by failure when connecting to redis itself.

[Hassaan Ahmed]

We've got more info on Stacktrace

StackExchange.Redis.RedisServerException: ERR value is out of range
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Akka.Persistence.Redis.Journal.RedisJournal.<ReplayMessagesAsync>d__5.MoveNext()

Also noticed a version difference, Azure is running Redis 3.0.7 while we're locally running 3.2.100 (where persistence works). Has Akka.Persistence.Redis been tested on version 3.0.7?

[Alex Valuyskiy]

@bannerflow-hassaan have you tried Akka.Persistence.Redis 0.2.0-beta or 0.2.5-beta?

[Hassaan Ahmed]

@alexvaluyskiy Updated to Akka.Persistence 1.1.2.30-beta and Akka.Persistence.Redis 0.2.0-beta but same error. Regarding Akka.Persistence.Redis 0.2.5-beta, strange that I'm able to select it from NuGet but after update Akka.Persistence.Redis.dll is removed from the project. When I checked folder \packages\Akka.Persistence.Redis.0.2.5-beta\lib\net45\ I only find an empty file named _._ but no dll. No error shown in Package Manager Console.