JobStats cleanup
QBui opened this issue · comments
I have a job that scheduled to run every 5 mins. After few weeks of job executions, the job stats get accumulated quite a few of entries, it caused the API api/v1/job/{id} to return lot of data. I am looking for ways to clean up these stats since I am only interested in checking for the last running job status.
I am thinking about changing kala code in db.go to add few more methods for cleaning up the old stats entries (e.g. removing items greater than 1 weeks old). I need some pointers to see if this is the right thing to do. I can contribute the code back once I am done.
Please let me know. Thanks,
-Quan
Approach #2: Removing stats items from the Cache and rely on JobCache.Persist() method to update the underlined database. This probably a quicker implementation. I will try out this approach.
Oh this is interesting. I think I would lean towards one of two solutions (or both of them):
- The ability to set a TTL for job stats, where after a certain amount of time specified by the user they are deleted. The default would be
neverfor backwards compatibility. - The ability to filter with the get request to only get N number of stats or stats in between a datetime range.
@QBui What do you think about these?
Thanks for the reply. I will start with the #1. It will address my issue and I also don't want to keep the old job stats around. IMO, the #2 is nice to have. It may require to implement additional REST resource, e.g. /api/v1/job/{id}/stats?since=. I am not sure about this use-case yet. I will revisit this one later.
@QBui Sounds great.
