[Incremental Ingestion Backend] Clean up entity marks after a few ingestions
taras opened this issue · comments
Taras Mankovski commented
Our mark and sweep strategy creates a mark for each committed entity. In the case of a large entity provider like LDAP with 100k+ users, that'll be 100k marks for every ingestion. We never clean up previous marks, which bloats the database unnecessarily.
One possible solution is to create a task that will run on an interval to delete marks from previous ingestions. Another option could be to delete older marks when computing deleted entities.