apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine

Home Page:https://datafusion.apache.org/ballista

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Implement a disk manager with a LRU cache to maintain caches on local disk

yahoNanJing opened this issue · comments

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

To manage the local file cache, a disk manager based on a LRU-based cache data structure will be introduced.

Describe the solution you'd like

The disk manager will be responsible for the following duties:

  • Load files from remote storage to local disk under a specific directory if they are not cached
  • Get the local file path for a remote file
  • Remove files on local disk when they are expired

Describe alternatives you've considered

Additional context