Is there any plan to provide an Java SDK for this cachelib ?
openinx opened this issue · comments
Hi guys.
I think the cachelib is a quite interesting libary which solves the combined RAM and SSD cache issue. I notice this interesting library because I'm considering how to maintain the recently queried dataset in the local SSD for speeding the Apache Spark queries. ( I think the commecial databricks products has implemented this interesting feature, please see [1]). Saying The spark's executor will maintain its chunks (can be 1MB or 512KB) into the local SSD once the query accessed the column data from remote object storage ( aws s3 or azure blobstore).
After reading the cachelib paper [2], I think this project may be the perfect open source candidate to maintain the cache chunks. So my questions are:
A. Is there any plan to provide an Java SDK for this library ? Or do you know any others has tried to accomplish that ? It's okay if nobody is currently working on this part, I think I can try to implement this Java SDK if we decided to choose cachelib finally :-)
B. Is there any plan to support compiling in MacOS ? It is just a tiny question, since seems our teams are currently using MacOS to develop the projects. It's okay if we don't have any plan to support MacOS, we can also accomplish the Java SDK in linux.
- https://docs.databricks.com/en/optimizations/disk-cache.html
- https://www.usenix.org/system/files/osdi20-berg.pdf
Best Regrads.
Hi @openinx: Thanks for reaching out!
A. We don't have a plan to provide Java SDK for now.
B. For MacOS compiling, the code is available at: https://github.com/agordon/CacheLib/tree/macos-v40 (note this is the macos-v40 branch, not “main” branch). Could you try the following and see if it works (it was introduced long time ago, so I'm not sure if it still works):
$ git clone --branch macos-v40 https://github.com/agordon/CacheLib.git
$ cd CacheLib
$ ./contrib/macos-01-brew.sh
Install dependencies:
$ ./contrib/macos-02-build-dependencies.sh
Build CacheLib:
$ ./contrib/macos-03-build-cachelib.sh
Got it, thanks @jiayuebao for the info.
The MacOS compiling links is really helpful !