Journal event deletion behavior difference
Arkatufus opened this issue · comments
In the current code as of november 2022, journal event deletion behavior in Persistence.Linq2Db is inconsistent with the rest of Persistence.Sql ecosystem.
- In Akka.NET, we handle deletion by storing the last deleted sequence number inside the metadata table. When determining the highest sequence humber, we do a query on both table and returns the highest number found.
- In JVM and Linq2Db, the last sequence number was not deleted, instead it is soft deleted and is excluded from queries, except when trying to determine the highest sequence number for the next insertion.
This difference needs to be rectified in order to make Persistence.Linq2Db consistent with the rest of the Sql ecosystem.
cc @to11mtm
@Aaronontheweb So this is one of the many vaunted compatibility mode switches
- The way JVM/Linq2Db do things is better because it minimizes the total number of pages written on commit.
- Also, it's one less page being read on read.
- FWIW, Persistence.Linq2db -should- be able to handle both.
- but to be clear it's kinda gnarly code, and there may be some edge cases around 'converting' to the JVM method.
I can say from experience that Linq2Db/JVM's approach handles frequent deletes -much- better than Sql.Common's JournalMetadata paradigm on MSSQL.
https://github.com/akkadotnet/Akka.Persistence.Linq2Db/blob/dev/src/Akka.Persistence.Sql.Linq2Db/Journal/DAO/BaseByteArrayJournalDao.cs#L385 <-- Compat Mode Query
https://github.com/akkadotnet/Akka.Persistence.Linq2Db/blob/dev/src/Akka.Persistence.Sql.Linq2Db/Journal/DAO/BaseByteArrayJournalDao.cs#L373 <-- JVM-style Native Query
We also handle it on the actual Delete of course, but the logic is a bit uglier.
This may or may not be another migration step to consider, or making this compat flag ternary for better forward-compatibility.
Thanks @to11mtm - this is very helpful. This issue was mostly created as a discussion point so we can reason about the backwards compat specs and what to do going forward.
Thanks @to11mtm - this is very helpful. This issue was mostly created as a discussion point so we can reason about the backwards compat specs and what to do going forward.
For sure; it's also worthwhile to have some level of documentation (even if it's in this issue) around the overall rationale for the change.
Realistically, a 'seamless' migration would require at minimum a migration script; the 'tricky' thing is that we would need some 'fake' event to put into the journal tables based on migration.
I suppose the other option would be the aforementioned ternary; it would at least minimize pain on 'new' persistenceIds (as they would elide at least the writes to Metadata) but would still have a read check+union (although, if metadata is properly indexed, should be fast to see it's not there.)
This would not be a problem if we remove the logical-delete and include-deleted flag from the HOCON settings. The fake entry would not be read during recovery and query operation because it is marked as deleted. It will break thing if users are actually reading the journal table directly though.
This would not be a problem if we remove the logical-delete and include-deleted flag from the HOCON settings. The fake entry would not be read during recovery and query operation because it is marked as deleted. It will break thing if users are actually reading the journal table directly though.
Any reason we shouldn't do this @to11mtm ? I think this approach makes a lot of sense.
My take is that we'll be implementing both model, but the metadata table will only be supported on a legacy system?
We will need to clean up the HOCON configuration for linq2db, its a bit too convoluted right now, especially with the addition of TagTable
@Arkatufus was this resolved in yesterday's PR? #145