Initial fetch of shadowserver-schema.json fails even in auto_update mode

[barbich]

hi
running latest intelmq via docker-compose and using latest shadowserver api and parser versions.

Observed behavior:

The shadowserver parser bot fails to start if the initial fetch of shadowserver-schema.json has not been done manually even if auto_update is set to True:

root@fe8423630f14:/opt# intelmqctl run Shadowserver-Parser process
Starting Shadowserver-Parser...
Shadowserver-Parser: ShadowserverParserBot initialized with id Shadowserver-Parser and intelmq 3.3.0 and python 3.9.2 (default, Feb 28 2021, 17:03:44) as process 5157. Standalone mode: True.
Shadowserver-Parser: Bot is starting.
Shadowserver-Parser: Loading source pipeline and queue 'Shadowserver-Parser-queue'.
Shadowserver-Parser: Connected to source queue.
Shadowserver-Parser: Loading destination pipeline and queues {'_default': ['File-Output-queue']}.
Shadowserver-Parser: Connected to destination queues.
Shadowserver-Parser: Bot initialization failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py", line 241, in __init__
    self.init()
  File "/usr/local/lib/python3.9/dist-packages/intelmq/bots/parsers/shadowserver/parser.py", line 61, in init
    config.reload()
  File "/usr/local/lib/python3.9/dist-packages/intelmq/bots/parsers/shadowserver/_config.py", line 339, in reload
    raise ValueError("The schema file does not exist: %r.", __config.schema_file)
ValueError: ('The schema file does not exist: %r.', '/opt/intelmq/var/lib/bots/shadowserver-schema.json')
Shadowserver-Parser: Bot stopped.
Bot exited with code 1.
Shadowserver-Parser: Bot stopped.
Exception ignored in: <function Bot.__del__ at 0x7fd6a6797af0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py", line 566, in __del__
    return self.stop(exitcode=0)
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py", line 599, in stop
    sys.exit(exitcode)
SystemExit: 0

Note:
intelmq.bots.parsers.shadowserver.parser --update-schema works fine

Expected behavior:

As per documentation "The parser will attempt to download a schema update on startup when the auto_update option is enabled"
Therefore it would be expected that with auto_enabled set to True the parser would download the schema file and move on (even if it has never been downloaded before).

This seems to be due to the fact that the parser raises a ValueError on line 339 instead of just logging an error message (IMHO).

Fix

• run intelmq.bots.parsers.shadowserver.parser --update-schema
• replace the error raise at line 339 by a __config.logger.error