Current time provider

[Jure]

On every lookup, item age is calculated and compared to the TTL. Internally, this results in a call to Stopwatch.GetTimestamp(), which is relatively expensive compared to the dictionary lookup that fetches the item.

Instead of calling Stopwatch.GetTimestamp() on every lookup, a current time provider object could be used that would hold a cached current time timestamp property. High frequency calls to get a current timestamp would become very cheap.

At initialization this provider would create a new thread/task that would update it's current time timestamp property at regular intervals. Interval's length would depend on time resolution passed into a constructor. For example, even a resolution of once per second would probably be enough for most purposes, and would make cost of calling Stopwatch.GetTimestamp() negligible.

class TimeProvider 
{
    public TimeProvider(TimeSpan resolution)    { ... } // creates and runs background thread (Task)

    // cached timestamp, updated in regular intervals by a background thread
    public long Timestamp { get; private set; }

    ...
}

NOTE: I haven't run any benchmarks to validate this. This is just an idea I had while reading the docs.

[Alex Peck]

Thanks for the idea - I will try coding this up and see how it could be integrated as an option.

Having a background thread is perhaps a matter of taste - I avoided spawning threads by default because it becomes more complicated to manage, especially for multiple caches - e.g. don't spawn 100 threads if 100 different caches are instantiated (this is a use case I depend on). The TimeProvider should be IDisposable to avoid leaking threads, so should the cache own the TimeProvider, which implies it is also IDisposable etc. There are a few details to resolve.

[Jure]

I think one instance of TimeProvider should used for multiple caches, unless different time resolutions are needed. But even then, it would probably make sense to create just one TimeProvider with a highest required time resolution (i.e. shortest interval length), as there is no harm, if TimeProvider is more precise than required for some caches.

[Ben Manes]

fwiw, some operating systems can read the time in userspace so that the overhead is just a few nanoseconds rather than making a system call. However, which variants are user-space is OS specific and may be impacted by virtualization. For example on macos the wall clock time is fast while nanotime slow, while on linux it is usually fast but the clock source on cloud is often slow due to the hypervisor (must be reconfigured). As timing is often used throughout the system (e.g. log events), if slow then the first step is to configure the system to default to a fast, user-space clock source.

Another caveat to be aware of is that historically getting the wall clock time was fast while the rtcs was slow, but since the wall clock is managed by the user it may drift and be reset by (e.g. by an ntp daemon). These days the drift is small in the cloud, but historically that could vary quite wildly in a managed network and is observable on mobile/iot devices. So you might want to default to prefer a high resolution clock and make it configurable (primarily for nicer unit testing).

[Alex Peck]

Thanks Ben, I hadn't considered the impact of a hypervisor. On Windows at least, the default clock impl isn't subject to drift/reset (see below).

With the current code the underlying clock impl can be configured by changing the time policy generic type arguments, but it's a bit fiddly - there is a note about how to do it here.

I included a policy based on each of the APIs available in the .NET framework, my knowledge of these is very windows centric:

• Environment.TickCount, which is based on GetTickCount() WinAPI function and has a precision of 15.6ms and about 1.5 ns overhead per call, but is represented by 32-bit unsigned integer (so would break after 49 days).
• DateTime.UtcNow, which is based on GetSystemTimeAsFileTime() WinAPI function and has a precision of 100ms and call latency of about 25ns. The system can periodically refresh the time by synchronizing with a time source, so this is subject to being reset etc.
• Stopwatch.GetTimestamp, which is based on QueryPerformanceCounter() WinAPI function, has a precision of <1us and a call latency of about 15ns. QPC is independent of, and isn't synchronized to, any external time reference. This is the default, since it gives stable results and is about 2x faster than DateTime.UtcNow.

On .NET Core onwards these will be mapped to different underlying macos and linux APIs - I will do some research on this and document it properly, it's totally possible I made a good choice for Windows but not others.

My feeling is that it is already fast enough - outside micro benchmarks the 15ns extra is likely noise. But I have a clean way to make this an option and it is simple to implement so I will at least try it out.

[Alex Peck]

There is a fourth option I had missed: .NET Core 3.0 added the Environment.TickCount64 property. This has the low latency of the 32-bit API, but will take 292,277,266 years to wrap around, so for practical purposes it will never wrap.

In PR #331 I switched the default clock to Environment.TickCount64 for .NET Core 3 and .NET6 build targets. With this change the overhead for the TLRU policy is very close to having time updated in the background (on Windows at least, this is effectively what is happening - TickCount64 is updated every 16ms).

[Jure]

An elegant solution. I like it, because it also keeps API simple.

I think there's no reason to keep this issue open anymore. Thank you!