Persistence AsyncWriteJournal is not realy async
kantora opened this issue · comments
I've encountered the very strange issue while executing the performance tests in my application.
I am using ClusterSharding and PersistentFSM for main actors, that represent business entities. For the storage, I use Akka.Persistence.Redis plugin (that is expected to be fast). Also, I configured TaskDispather as default one.
Unfortunately, performance was much lower than I expected. Investigation showed that every actor in the system had empty message queue except for /system/akka.persistence.journal.redis with more than hundred messages (I am still working, but as for now, entity actors spawn rather large amount events to persist). Every time I measured it was always WriteMessages in the process.
That is strange, as it seems to me from code that AsyncWriteJournal.HandleWriteMessages should spawn async task and proceed to other messages. So it should be fast.
I used dotTrace to look at it a little bit closer.
Now I am sure that everything is processed in the strictly single thread, So all my multithreaded application is depended on it as all actors are waiting their queue to persist.
@kantora Try the newest version of Redis plugin on nuget. But this version is incompatible with the previous one, because I've changed the structure of snapshot entity.
Sorry for late response.
I've checked it - now, according to dotTrace it is working in two threads. but definitely as separate tasks. So everything seems to be ok now. Thanks.