Giters

akkadotnet / Akka.Persistence.Sql

Linq2Db implementation of Akka.Persistence.Sql. Common implementation for SQL Server, Sqlite, Postgres, Oracle, and MySql.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Improve Persistence.Query Memory Usage

to11mtm opened this issue · comments

commented

The current Implementation of Persistence.Query may not scale well on certain queries.

The biggest problem is that we are using .ToList() in places that in large systems may return large results.

We have two options. The first of which is a 'universal' fix and can easily be implemented as an opt-in feature, the second will require a little more work but will be better for users long term.

  1. Change Persistence.Query methods that call .ToListAsync() to instead stream off an IAsyncEnumerable, via https://gist.github.com/to11mtm/dc9a350080fcbcb14098c14509d70e7f
  2. Refactor methods that can use the 'batched chunk' read style used by MessagesWithBatch

Option 1 is the 'safest' and should work with all DBs with the possible exception of SQLite (because of it's tendencies for readers and writers to block each other.) Option 2 will be better performing.

I'm expecting that the end state will be that most queries will be better under 'batched' reads, but there will be one or two (I'm thinking mainly CurrentPersistenceIds()) that will still benefit from the option of switching.