twitter / pelikan

Pelikan is Twitter's unified cache backend

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Refactor datapool related code

thinkingfish opened this issue · comments

  • move datapool into storage module;
  • allow shm/pmem backed storage to be compiled simultaneously, into different binaries;
  • item_relink() can be slightly refactored;
  • do another round of name checks to see if any names can be made more intuitive

Even though I came up with it, in hindsight I think that the name shm doesn't fully represent what datapool_shm does. A more descriptive name would be anon for anonymous memory (as opposed to file backed memory), but that's really an implementation detail. Maybe datapool_vmem so that we will have volatile memory and persistent memory backends?

Also, notice that right datapool_pmem will fallback to malloc if no path is provided. I did that to avoid excessive abstraction, but it would be cleaner to simply fallback to datapool_shm in that case.

I think we coud have a real shared memory implementation instead of anonymous memory, using shared memory can provide benefits like graceful restart, which is needed in some use cases such as updating Pelikan without loss of cached contents. Moreover, we can have a unified interface for shm and pmem implementation.
Although I am not sure whether shared memory can be supported by mesos or not.