Massive CPU spike from threads not completing
johnscancella opened this issue · comments
Hello,
I am seeing a massive CPU spike when I install the plugin (near 100% CPU utilization). I noticed that there are many prometheus threads running in the background continuously (20+). After uninstalling the plugin it drops back down to idle.
Thanks for the feedback @johnscancella! Would you mind filling this issue under https://github.com/jenkinsci/prometheus-plugin
And would be great if you can give me info about how many jobs / builds you have?
do you have any pipline-multibranch jobs (former workflow-plugin)?
I iterate over all builds from each job, might be possible that it gets stuck somewhere in this loop:
https://github.com/jenkinsci/prometheus-plugin/blob/master/src/main/java/org/jenkinsci/plugins/prometheus/JobCollector.java#L81
I'm running the plugin myself but I only have multibranch jobs.. and I dont see the same issue.
I would be happy to file it under that project, but it doesn't list the issues tab or a button to create an issue.
We have a bunch of jobs, some of them the new pipeline jobs (2). But even disabling almost all the jobs did not fix anything.
Now that I am back at work here are the specifics:
42 jobs (2 are pipeline jobs)
384 total builds
I also created a pull request that has a bunch of logging added so it should be much easier to see what is actually happening
I rebuilt the plugin using the head of Master and it seems to be working fine. My coworker @acdha found that the line run = run.getPreviousBuild(); was added jenkinsci/prometheus-plugin@prometheus-1.0.5...master which prevents a infinite loop if the first run is a workflowRun and the execution is null.
So simply releasing a new version with the current code should fix it. Though it would still be nice to have the debug logging.
cool! thanks! will do new release soonish!
@lsjostro I see I can now create issues on https://github.com/jenkinsci/prometheus-plugin/issues. Did you want me to re-file this issue over there?