RuntimeError: dictionary keys changed during iteration
heiner0815 opened this issue · comments
Hi,
after upgrading to version 0.9.1 the application fails with the following exception (see below)
In fact, it seems that since python 3.8 code like the following:
for key in keys:
drain.id_to_cluster[int(key)] = drain.id_to_cluster.pop(key)
is no longer valid and leads to the observed exception (when starting the parser for the second time and loading the pickeled state)
...
File "/home/wollny/.cache/pypoetry/virtualenvs/src-k3pL4lLu-py3.8/lib/python3.8/site-packages/drain3/template_miner.py", line 56, in __init__
self.load_state()
File "/home/wollny/.cache/pypoetry/virtualenvs/src-k3pL4lLu-py3.8/lib/python3.8/site-packages/drain3/template_miner.py", line 74, in load_state
for key in keys:
RuntimeError: dictionary keys changed during iteration
Would be nice if you could fix this, as I like your project very much :-)
PS: It would be also nice if you could upgrade the jsonpickle package, as version 1.4.1 has a known safety issue, which got fixed in version 1.4.2
nox > safety check --file=requirements.txt --full-report
+==============================================================================+
| |
| /$$$$$$ /$$ |
| /$$__ $$ | $$ |
| /$$$$$$$ /$$$$$$ | $$ \__//$$$$$$ /$$$$$$ /$$ /$$ |
| /$$_____/ |____ $$| $$$$ /$$__ $$|_ $$_/ | $$ | $$ |
| | $$$$$$ /$$$$$$$| $$_/ | $$$$$$$$ | $$ | $$ | $$ |
| \____ $$ /$$__ $$| $$ | $$_____/ | $$ /$$| $$ | $$ |
| /$$$$$$$/| $$$$$$$| $$ | $$$$$$$ | $$$$/| $$$$$$$ |
| |_______/ \_______/|__/ \_______/ \___/ \____ $$ |
| /$$ | $$ |
| | $$$$$$/ |
| by pyup.io \______/ |
| |
+==============================================================================+
| REPORT |
| checked 78 packages, using default DB |
+============================+===========+==========================+==========+
| package | installed | affected | ID |
+============================+===========+==========================+==========+
| jsonpickle | 1.4.1 | <=1.4.1 | 39319 |
+==============================================================================+
| Jsonpickle through 1.4.1 allows remote code execution during deserialization |
| of a malicious payload through the decode() function. See CVE-2020-22083. |
+==============================================================================+
I could not reproduce the RuntimeError: dictionary keys changed during iteration issue under Python 3.8 (added a CI build with Python 3.8 which did not raise this exception). However, for the robustness of the code, I did some refactoring to avoid the str-to-int type conversion on load at all. This should solve it.
jsonpickle also updated to v1.5.1.
For what it's worth, persistence lead to this exact error on v0.9.1 (using the examples to make it easier to reproduce):
- On Python 3.9.1, persistence to a file lead to this result.
Starting Drain3 template miner
Loading configuration from drain3.ini
config file not found: drain3.ini
Checking for saved state
Traceback (most recent call last):
File "/home/sayrus/Workspace/PERSO/drain3/examples/drain_stdin_demo.py", line 43, in <module>
template_miner = TemplateMiner(persistence)
File "/home/sayrus/.local/share/virtualenvs/drain3-sC6cDVeE/lib/python3.9/site-packages/drain3-0.9.1-py3.9.egg/drain3/template_miner.py", line 62, in __init__
File "/home/sayrus/.local/share/virtualenvs/drain3-sC6cDVeE/lib/python3.9/site-packages/drain3-0.9.1-py3.9.egg/drain3/template_miner.py", line 79, in load_state
- On Python 3.8.7, it works
- On Python 3.8.0, it works
Do you get this error also on v0.9.2?
It is fixed on v0.9.2 !
(Editing the previous message, I did not update the dependency to check on the new version)