Job runs multiple times in the same schedule
rbiq opened this issue · comments
I'm having a really weird issue with your library. In development the hosted service job on an asp.net core 3.1 web application will be called only once as set in cron schedule (once a day at hh:mm:ss A/PM). However when deployed to production on aws EC2 it runs multiple times. And the number is also arbitrary and not fixed. So like 1 day it will run 5 times, next week 8 and so on. Have you heard of an issue like that? I'm not sure if it could be related to number of cpu cores or something else. However this is happening only in production which is really bothersome. I like your library and want to use it however I'm currently forced to remove and replace it with something else to see if the issue still persists. I was only wondering if you had any similar issue you came across or have someone report to you about it. Please let me know as I'm eager to find a solution to this problem using your library which I really like and appreciate so much.
Here's the code in case you're interested:
public abstract class ScheduledBackgroundWorker<T> : BaseBackgroundWorker<T> where T : class
{
private CrontabSchedule schedule;
private DateTime nextRun;
protected string JobName => this.GetType().Name;
protected abstract string JobDescription { get; }
protected abstract string Schedule { get; }
protected ScheduledBackgroundWorker(IServiceProvider services,
ILogger<T> logger) : base(services, logger)
{
schedule = CrontabSchedule.Parse(Schedule, new CrontabSchedule.ParseOptions
{
IncludingSeconds = true
});
nextRun = schedule.GetNextOccurrence(DateTime.Now);
}
protected abstract Task ExecuteOnStartAsync(CancellationToken cancellationToken);
protected abstract Task ExecuteOnStopAsync(CancellationToken cancellationToken);
protected override async Task ExecuteStartAsync(CancellationToken cancellationToken)
{
logger.LogInformation("{jobName} Schedule {schedule} was scheduled and next run will be at {nextRun} - {now}", JobName, Schedule, nextRun.ToString("MM/dd/yyyy hh:mm:ss t"), DateTime.Now);
using var scope = services.CreateScope();
var cache =
scope.ServiceProvider
.GetRequiredService<IDistributedCache>();
do
{
var hasRun = await cache.GetAsync(JobName);
if (hasRun == null)
{
var now = DateTime.Now;
if (now > nextRun)
{
logger.LogInformation("{jobName} Schedule {schedule} is running - {now}", JobName, Schedule, DateTime.Now);
await ExecuteOnStartAsync(cancellationToken);
nextRun = schedule.GetNextOccurrence(DateTime.Now);
var currentTime = nextRun.ToString();
byte[] encodedCurrentTime = Encoding.UTF8.GetBytes(currentTime);
var options = new DistributedCacheEntryOptions().SetAbsoluteExpiration(TimeSpan.FromSeconds(60));
await cache.SetAsync(JobName, encodedCurrentTime, options, cancellationToken);
logger.LogInformation("{jobName} Schedule {schedule} has finished and next run will be at {nextRun} - {now}", JobName, Schedule, nextRun.ToString("MM/dd/yyyy hh:mm:ss t"), DateTime.Now);
}
}
await Task.Delay(5000, cancellationToken); //5 seconds delay
}
while (!cancellationToken.IsCancellationRequested);
await Task.CompletedTask;
}
protected override async Task ExecuteStopAsync(CancellationToken cancellationToken)
{
await ExecuteOnStopAsync(cancellationToken);
logger.LogInformation("{jobName} Schedule {schedule} was stopped - {now}", JobName, Schedule, DateTime.Now);
await Task.CompletedTask;
}
public override void Dispose()
{
}
}
Even tried to control it using cache but no luck.
public class DummyTestJob : ScheduledBackgroundWorker<DummyTestJob>
{
public DummyTestJob(IServiceProvider services,
ILogger<DummyTestJob> logger) : base(services, logger)
{
}
protected override string JobDescription => "Dummy Test";
protected override string Schedule => "10 * * * * *";
protected override async Task ExecuteOnStartAsync(CancellationToken cancellationToken)
{
using var scope = services.CreateScope();
var apiClient =
scope.ServiceProvider
.GetRequiredService<IApiClient>();
var runtimeEnvironment = scope.ServiceProvider.GetRequiredService<IRuntimeEnvironment>();
try
{
var page = 1;
var size = 100;
var loop = true;
List<User> users = new List<User>();
do
{
try
{
var response = await apiClient.GetAsync($"users?page={page}&size={size}", null, null);
if (response.Count > 0)
{
users.AddRange(response.Data);
page++;
}
else
{
loop = false;
}
}
catch (Exception ex)
{
logger.LogError(ex, "Failure getting users for page {page}", page);
loop = false;
}
} while (loop);
List<string> userNames = new List<string>();
foreach (var user in Users)
{
if (user.IsActive == 1)
{
userNames.Add(user.Name);
}
}
users.Clear();
users = null;
foreach (var userName in userNames)
{
System.Console.WriteLine($"Hello {userName}...\n");
}
}
catch (OperationCanceledException ex)
{
logger.LogError(ex, "{JobName} execution canceled - {now}", JobName, DateTime.Now);
}
await Task.CompletedTask;
}
protected override async Task ExecuteOnStopAsync(CancellationToken cancellationToken)
{
await Task.CompletedTask;
}
}
I was only wondering if you had any similar issue you came across or have someone report to you about it.
I am afraid not.
I'm currently forced to remove and replace it with something else to see if the issue still persists.
Let me know what you find out and I hope you get to the bottom of your issue. NCrontab is a very simple and barebones library so I'd hate to say it, but I think it seems to have something to do with your production setup or environment. NCrontab has been around for 12 years, downloaded nearly 30 millions times, and powers timer triggers in Azure Functions on Microsoft's cloud so if there way a fundamental problem like you seem to be experiencing, I am sure I would have heard of it by now.
Did you get a chance to review the code and see if there was anything wrong I was doing there? Also can you please provide a few examples on how you would use your library in a production environment like in a hosted service for example? That would be really helpful.
Did you get a chance to review the code and see if there was anything wrong I was doing there? Also can you please provide a few examples on how you would use your library in a production environment like in a hosted service for example?
I don't have that kind of time. This is free and open source software. It is a relatively small and featureless library that does two things: parse crontab expressions and calculate the next occurrence of a schedule. It should be relatively easy to study or debug if you think the problem is with the library though I would first try to reproduce the problem in development. Other than that, you seem to have enough logging in there to inspect what's going on at runtime.
Thanks. Best wishes.
@atifaziz I figured out the issue and have reinstalled your fantastic library. It was related to IIS Max Worker Process setting. Thank you so much for your help!