ConcurrentTimedRotatingFileHandler Deletes wrong files

[Eric]

The behavior changed after the last updated, using python 3.9.15

Previously backupCount: 5 would keep 5 days worth of logs

if today is 2023-12-19 the logs would look like

mylog.log
mylog.log.2023-12-18.gz
mylog.log.2023-12-18.1.gz
mylog.log.2023-12-18.2.gz
mylog.log.2023-12-17.gz
mylog.log.2023-12-17.1.gz
mylog.log.2023-12-17.2.gz
mylog.log.2023-12-16.gz
mylog.log.2023-12-16.1.gz
mylog.log.2023-12-16.2.gz
mylog.log.2023-12-15.gz
mylog.log.2023-12-15.1.gz
mylog.log.2023-12-15.2.gz
mylog.log.2023-12-14.gz
mylog.log.2023-12-14.1.gz
mylog.log.2023-12-14.2.gz
mylog.log.2023-12-13.gz
mylog.log.2023-12-13.1.gz
mylog.log.2023-12-13.2.gz

Now it is only keeping 5 backups, and i'm guessing its based on last modified date so we end up with something like this, where we only have 5 backups and mylog.log.2023-12-17.1.gz was deleted because of the order it was renamed i guess

mylog.log
mylog.log.2023-12-18.gz
mylog.log.2023-12-18.1.gz
mylog.log.2023-12-18.2.gz
mylog.log.2023-12-17.gz
mylog.log.2023-12-17.2.gz

i haven't dug into this too much, just pointing it out.

[Preston Landers]

I guess the question would be how does the base Python TimedRotatingFileHandler handle this. The docs say:

If backupCount is nonzero, at most backupCount files will be kept, and if more would be created when rollover occurs, the oldest one is deleted. The deletion logic uses the interval to determine which files to delete, so changing the interval may leave old files lying around.

So yeah, it seems like this is the expected behavior for the base class. I think you would only get the multiple files per day if you have a size limit (maxBytes).

[Eric]

I think what it was doing before was more inline what i thought it was supposed to be doing.

If backupCount=5, maxBytes=100mb
If we output 1gb of logs per day, we would end up with 5 days worth of logs, and 10x100mb files for each day

I get that maxBytes limits the size, but i figured in the case of timed rotation, the backup count would be the number of days backed up, and not the number of files

[Preston Landers]

The intention was to behave similar to the parent class (TimedRotatingFileHandler), and it seems relatively clear to me that "backupCount=5" means that five backup files should be kept, not necessarily "5 intervals". In fact, the decision about what files to delete is made in the parent class getFilesToDelete. But one difference is that the parent class doesn't support limiting the file size, which can produce multiple files per interval.

I don't want to change the behavior of backupCount in that regard. However I would be open to a PR which adds a separate option (maybe backupCountByInterval) that could override that and provide the requested behavior.

[Eric]

That makes sense. I think backupCountByInterval would be a great addition!