Strategy for Meter & ActionScheduler when schedules cannot be served at requested intervals

[Helge Jensen]

I am missing a strategy for what to do when ActionScheduler cannot meet the requested interval between invocations.

I am using the (very nice) Metrics.NET library to monitor thousands of Meters. Unfortunately that means that the Meters default Tick() refresh-rate at MeterMetric.TickInterval = 5s cannot be serviced.

As a result, the ThreadPool spins up more Threads, which actually makes the problem worse, by causing a cascade of Thread creations. Eventually upto ThreadPool.MaxThreads and consumes all OS resources changing between threads that update Meters :(

The below code demonstrates the issue (on my machine, milage may vary as it depends on slowness of Meter.Tick()-updates).

        [Fact]
        public void ActionScheduler_ShouldNotCascadeThreadCreationWhenManyMetersAreInstantiated()
        {
            var count = 50000;
            var meters = Enumerable.Range(0, count)
                .Select(i =>
                {
                    var meter = Metric.Meter(string.Format("M{0}", i), Unit.Calls, TimeUnit.Seconds);
                    meter.Mark();
                    return meter;
                })
                .ToList();
            // The mere *existence* of many meters makes the ActionScheduler schedule a large
            // amount of tasks that cannot execute at the desired (5s) intervals on some machines
            // 1. leading to the TaskPool extending the number of threads, 
            // 2. leading to even more scheduling time spent, and even worse execution (goto 1)
            Func<int> threadCount = () => System.Diagnostics.Process.GetCurrentProcess().Threads.Count;
            var threadCountInitial = threadCount();
            Thread.Sleep(TimeSpan.FromSeconds(20));
            var threadCountAfterSomeTime = threadCount();
            Assert.True(threadCountAfterSomeTime <= threadCountInitial + 1);
        }

I am using a workaround that extends the ActionScheduler with a TaskScheduler to run the task on, defaulting to TaskPool.Default (with exactly the old behaviour). But I can inject a thread-limited TaskScheduler via a static default to use for the Scheduler. This means that I can limit the amount of effort dedicated to Meter updates and live with the slow update-times that gives.

[Helge Jensen]

Since nobody else has this issue, and I stopped using Metrics.net, I am closing this issue as not-planned