Question about obtaining the benchmark result
cherishwsx opened this issue · comments
Thank you for all the amazing work you've done!
I successfully ran through the training and predicting process of deeplog model using the same HDFS data file that you are using (from loghub).
And I'm using Drain as my parsing tool to get the structured log data. I ended up having 48 unique event ID in the template. And I'm using around 5000 sessions for the training and the train loss and validation loss converged to 0.2 (start from 0.8) around 300+ epochs. I didn't change the default parameter setting in the deeplog.py file except for the number of classes (48 in my case).
The result that I got from prediction is shown below. It does not look as promising as the benchmark.
I'm not sure why but is it because of the parsing tool?
And idea or suggetions of improving the model results are welcome!!
And forgot to ask, could you breifly explain what is the num_candidates parameter for in the prediction?
Thank you!!!!
It depend on your parsing tool, my benchmark result is depend on "the ground truth" number of the template(28) in dataset"
num_candidates means the label in top num_candidates is labeled as normal log.
(you need to read the deeplog paper to get a better understanding of num_candidates...)
- try to finetune num_candidates to get a better F1 score.
- try to modify your parsing code to get a result close to the Ground truth(28 templates)
Thank you so much for the suggestions! That's really helpful!
One follow up question I have is that, this may sounds a naive question, but do we always know the ground truth number of the log? And when we are using the parsing tool, we want to have the result/template as close as possible to the ground truth number we know by modifying the parsing code?
In industrial applications, the constantly updated log has no definite ground truth templates, you need to continuously optimize the model based on performance indicators :)
Got it! Thank you! I don't have further question for now! :))