Code has moved to https://github.com/busterwood/Goodies
A low memory overhead read-through cache implemented using generations.
The design is layered so you can choose the functionality that you require:
Cache<TKey, TValue>
is a thread-safe generational cache-aside class that you populate with data.ConcurrentCache<TKey, TValue>
extendsCache<TKey, TValue>
to support concurrent reading and writing.ReadThroughCache<TKey, TValue>
is a thread-safe generational cache that populates itself from an underlying data source.BatchReadThroughCache<TKey, TValue>
extendsReadThroughCache<TKey, TValue>
adding support for reading multiple values for multiple keys in one call.ThunderingHerdProtection<TKey, TValue>
prevents concurrent read-through for the same key (see below).BatchThunderingHerdProtection<TKey, TValue>
prevents concurrent read-through for multiple keys (see below).
All caches implementes the basic the ICache<TKey, TValue>
interface, which has methods such as:
TValue this[TKey] { get; set; }
sets or tries to get a value for a key (returnsdefault(TValue)
is not found, i.e. null)Task<TValue> GetAsync(TKey key)
tries to get a value for a key asynchronuslyint Count()
returns the number of items currently in the cachevoid Clear()
removes all items from the cachevoid Remove(TKey key)
evicts a specific key from the cache
Batch loading classes implement the IBatchCache<TKey, TValue>
interface, which extends ICache<TKey, TValue>
adding the following methods:
TValue[] GetBatch(IReadOnlyCollection<TKey> keys)
tries to get the values associated with some keysTask<TValue[]> GetBatchAsync(IReadOnlyCollection<TKey> keys)
tries to get the values associated with the keys asynchronusly
You can also compose caches with the following extension methods:
WithGenerationalCache(int? gen0Limit, TimeSpan? timeToLive)
create a new read-through cache that has a Gen0 size limit and/or a periodic collection timeWithThunderingHerdProtection()
adds ThunderingHerdProtection to a cache which prevents calling the data source concurrently for the same key.
ThunderingHerdProtection<TKey, TValue>
can be used to prevent multiple threads calling an underlying database, or remote service, to load the value for the same key.
Different keys are handled concurrently, but indiviual keys are read by only one thread. This can be used on the client side, before calling a remote service or database,
and it can be used on the the server side.
The design is insipred by "generational garbage collection" in that:
- the cache as two generations
Gen0
andGen1
- as items are read from the underlying data source they are added to
Gen0
- when a collection happens
Gen1
is thrown away andGen0
is moved toGen1
- when an item is read from
Gen1
it is promted back toGen0
Internally, Cache<TKey, TValue>
uses a lock (Monitor) which is exposed as the SyncLock
property. ReadThroughCache<TKey, TValue>
releases the lock if a read to the IDataSource<TKey,TValue>
is required, reacquiring the lock to update the cache after the read.
The ReadThroughCache<TKey, TValue>
contructor takes two arguments that control collection:
- if a
gen0Limit
is set then a collection will occur whenGen0
reaches that limit - if
timeToLive
is set then a an un-read entry will be evicted some time after this (unless thegen0Limit
was reached since the last collection)
One or both parameters neeed to be set, i.e.
- you can set a
gen0Limit
, but then the cache might never be cleared for a long time if thegen0Limit
is never reached. - you can set a
timeToLive
which will let the cache grow to any size but will ensure items not used for "a long time" are evicted - you can specify both
gen0Limit
andhalfLife
to combine the attributes of both
ReadThroughCache<TKey, TValue>
remembers the results of all read-though operations, so it records the value for a key or it records that a key does not have a value.
The following tests compare a generation cache with a Bit-Pseduo LRU cache.
Test | Elapsed | Items in cache | Bytes allocated | Bytes held | Bytes held per key |
---|---|---|---|---|---|
BitPseudoLru 50% | 118 ms | 50,000 | 2,973,144 | 1,519,864 | 30.40 |
Generational 25% Gen0 Limit | 34 ms | 33,332 | 3,782,700 | 895,764 | 26.87 |
Generational 50ms Half-life | 25 ms | 100,000 | 6,050,996 | 3,129,228 | 31.29 |
Concurrent Dictionary | 11 ms | 100,000 | 5,651,692 | 2,994,008 | 29.94 |
Test | Elapsed | Items in cache | Bytes allocated | Bytes held | Bytes held per key |
---|---|---|---|---|---|
BitPseudoLru 50% | 2,062 ms | 250,000 | 9,686,520 | 6,529,452 | 26.12 |
Generational 25% Gen0 Limit | 183 ms | 166,664 | 11,926,556 | 4,637,388 | 27.82 |
Generational 50ms Half-life | 159 ms | 331,819 | 31,131,248 | 9,617,896 | 28.99 |
Concurrent Dictionary | 205 ms | 500,000 | 37,960,620 | 16,607,692 | 33.22 |