aleph-im / aleph-indexer-framework

TypeScript framework for deploying distributed indexers on Aleph VMs for Solana and EVM chains.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Optimize DB usage & size

MHHukiewitz opened this issue · comments

There are multiple reasons why the indexing framework is using too much disk space right now:

  • Data is being stored as serialized UTF-8 chars of its JSON representation. This could be replaced by ProtoBuf serialized data.
  • The EntityRequestResponseDAL stores all responses indefinitely. Freeing up space there would bring the biggest benefits but is not trivial as expected. Multiple indexers can request the same events, so we need to only delete these EntityRequestResponses if all Requests have been processed and satisfied. This might be solved by introducing a new job that periodically checks the DB for these finished Requests and deletes their Responses. Solved here: #34