logpai / Drain3

A robust streaming log template miner based on the Drain algorithm

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

PermissionError when running with Persistance

MarinSL opened this issue · comments

I am running drain on multiple log files with persistence, as I want to later use the state to match newer logs.
However, after a random running time (varying from ~10 mins to ~80 mins), I get a Permission error:

...
Saving state of 1024 clusters with 85804 messages, 248088 bytes, reason: cluster_template_changed (7971)
Saving state of 1024 clusters with 86332 messages, 248052 bytes, reason: cluster_template_changed (7276)
Saving state of 1024 clusters with 86612 messages, 248088 bytes, reason: cluster_created (8046)
Saving state of 1024 clusters with 86668 messages, 248000 bytes, reason: cluster_template_changed (7996)
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
Cell In[146], line 3
      1 train_csv= 'passed.csv'
      2 for f in filepaths:
----> 3     read_data(f,train_csv)

Cell In[140], line 44, in read_data(location, csv_location, drain, header)
     42             # Add to drain
     43         if drain:
---> 44             template_miner.add_log_message(message_cleaned)
     46 for row in rows:
     47         csv_writer.writerow(row[:-1])

File c:\Users\user\AppData\Local\Programs\Python\Python311\Lib\site-packages\drain3\template_miner.py:141, in TemplateMiner.add_log_message(self, log_message)
    139 snapshot_reason = self.get_snapshot_reason(change_type, cluster.cluster_id)
    140 if snapshot_reason:
--> 141     self.save_state(snapshot_reason)
    142     self.last_save_time = time.time()
    143 self.profiler.end_section()

File c:\Users\user\AppData\Local\Programs\Python\Python311\Lib\site-packages\drain3\template_miner.py:107, in TemplateMiner.save_state(self, snapshot_reason)
    102     state = base64.b64encode(zlib.compress(state))
    104 logger.info(f"Saving state of {len(self.drain.clusters)} clusters "
    105             f"with {self.drain.get_total_cluster_size()} messages, {len(state)} bytes, "
...
   1042 if "b" not in mode:
   1043     encoding = io.text_encoding(encoding)
-> 1044 return io.open(self, mode, buffering, encoding, errors, newline)

PermissionError: [Errno 13] Permission denied: 'drain3_state.bin'

The drain state file is not empty, so it seems to be able to access it most of the time.

I set up drain this way:

config = TemplateMinerConfig()
config.load("drain3.ini")

persistence = FilePersistence("drain3_state.bin")

template_miner = TemplateMiner(persistence, config)

logger = logging.getLogger(__name__)
logging.basicConfig(stream=sys.stdout, level=logging.INFO, format='%(message)s')

I am running it using Jupyter extension for Visual Code, if it helps

UPD: Works fine if ran by opening Jupyter Notebook directly, so seems to be a Visual Code thing