thefrontside / playhouse

Frontside's Backstage Portal

Home Page:https://backstage.frontside.services/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Incremental Ingestion Backend] Clean up entity marks after a few ingestions

taras opened this issue · comments

Our mark and sweep strategy creates a mark for each committed entity. In the case of a large entity provider like LDAP with 100k+ users, that'll be 100k marks for every ingestion. We never clean up previous marks, which bloats the database unnecessarily.

One possible solution is to create a task that will run on an interval to delete marks from previous ingestions. Another option could be to delete older marks when computing deleted entities.