Optimize DB usage & size
MHHukiewitz opened this issue · comments
There are multiple reasons why the indexing framework is using too much disk space right now:
- Data is being stored as serialized UTF-8 chars of its JSON representation. This could be replaced by ProtoBuf serialized data.
- The
EntityRequestResponseDAL
stores all responses indefinitely. Freeing up space there would bring the biggest benefits but is not trivial as expected. Multiple indexers can request the same events, so we need to only delete theseEntityRequestResponse
s if all Requests have been processed and satisfied. This might be solved by introducing a new job that periodically checks the DB for these finished Requests and deletes their Responses. Solved here: #34