Optimize for size-bounded cache

[Ercan Ozturk]

Currently we have a timed cache with unbounded space. Lifetime of the items in the cache is 30 seconds.

With the growing tree size and increasing number of requested operations (e.g., lookups), the size of the cache may become a limiting factor for how many operations we can handle. In that case, we will need to manage the cache more effectively.

To this end, some initial ideas are:

• Varying expiration times for nodes depending on their use frequency.
• Prioritize some (e.g., most used?) nodes during batch_get operations.

cc @afterdusk

[Kevin Lewi]

I believe this was resolved by #269, @eozturk1 can you confirm?

[Ercan Ozturk]

Adding support for memory pressure processing to limit the cache size based on the memory pressure on the system

Yes!