Giters

logpai / Drain3

A robust streaming log template miner based on the Drain algorithm

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

QUESTION: How to update previous results after 'cluster_template_changed'

stianvale opened this issue · comments

commented

Hi!
Great repository, thanks for putting it out there!

I have a quick question; do you have built-in logic for updating the previous results whenever the cluster template changes? Just want to verify this before I start making my own logic on it.

Best regards,
stianvale

commented

Hello,
What do you mean by "previous results"?
Can you provide an example of your need?

commented

Hi @davidohana, and thanks for replying!

I'm using Drain for feature generation for a machine learning case. I have a dataframe of log messages that I run through Drain to extract the patterns and categorize log messages. I then add these features to the dataframe.

Let's say I have one row with the log message 'aa aa aa' and then later on 'aa aa ab' as in your example in the readme.
When Drain sees 'aa aa ab' after having seen 'aa aa aa', it learns that the template is actually 'aa aa <*>'.

My preprocessing of 'aa aa aa' will still hold the template 'aa aa aa', but what I want to do now is to update this with the new template.

I could of course keep track of the cluster_id's and just update them manually after every 'cluster_template_changed', but I'm just wondering if there's some built-in logic in this library that can enable me to take care of this more efficiently.

Does that make sense?

commented

It is recommended to track clusters by the cluster ID and not the template, exactly because of this issue.

Look at the following sample output:

Starting Drain3 template miner
Checking for saved state
Saved state not found
Drain3 started with 'FILE' persistence
reading from std-in (input 'q' to finish)
aa aa aa
Saving state of 1 clusters with 1 messages, 952 bytes, reason: cluster_created (1)
{"change_type": "cluster_created", "cluster_id": 1, "cluster_size": 1, "template_mined": "aa aa aa", "cluster_count": 1}
parameters: []
aa aa ab
Saving state of 1 clusters with 2 messages, 1052 bytes, reason: cluster_template_changed (1)
{"change_type": "cluster_template_changed", "cluster_id": 1, "cluster_size": 2, "template_mined": "aa aa <:*:>", "cluster_count": 1}
parameters: ['ab']
aa aa cc
{"change_type": "none", "cluster_id": 1, "cluster_size": 3, "template_mined": "aa aa <:*:>", "cluster_count": 1}
parameters: ['cc']

The cluster_id stays 1 but template_mined changes.
Indeed if you save some repository of cluster ID to template mapping, you can update it after each cluster_template_changed event.

BTW this is one of the differences from the original Drain. which identifies clusters by the hash code of the template ID.

commented

Thanks a lot @davidohana, that perfectly explains it! If I would need the 'template_mined' further, then I'll just update the previous entries based on their 'cluster_id'.

Closing the question, thanks a lot!