Shadowserver parser may fail to autodetect report type after reload

[kamil-certat]

Handling feed detection

During the init method the parser try to determine the feed type detection type [0]:

intelmq/intelmq/bots/parsers/shadowserver/parser.py

Lines 42 to 43 in 7674949

 def init(self): 
     if self.feedname is not None: 

If the self.feedname isn't set, then the mode is detect [1]:

intelmq/intelmq/bots/parsers/shadowserver/parser.py

Lines 53 to 54 in 7674949

 else: 
     self._mode = 'detect' 

Later, during processing a report, the feed name is detected and eventually set [2]:

intelmq/intelmq/bots/parsers/shadowserver/parser.py

Lines 76 to 79 in 7674949

 if not retval: 
     raise ValueError('Could not get a config for {!r}, check the documentation.' 
                      ''.format(self.report_name)) 
 self.feedname, self._sparser_config = retval 

Reloading bot

Then, during the reload (not restart) happens, the bot is re-initialized without creating a new object:

intelmq/intelmq/lib/bot.py

Lines 271 to 285 in 7674949

     def __handle_sighup(self): 
         """ 
         Handle SIGHUP. 
         """ 
         if not self.__sighup.is_set(): 
             return False 
         self.logger.info('Handling SIGHUP, initializing again now.') 
         self.__disconnect_pipelines() 
         try: 
             self.shutdown()  # disconnects, stops threads etc 
         except Exception: 
             self.logger.exception('Error during shutdown of bot.') 
         self.logger.handlers = []  # remove all existing handlers 
         self.__sighup.clear() 
         self.__init__(self.__bot_id_full, sighup_event=self.__sighup) 

This mechanism is used during log rotation by our standard Debian configuration:

intelmq/contrib/logrotate/intelmq

Line 12 in 7674949

 sudo -u intelmq /usr/local/bin/intelmqctl --quiet reload 

and of course re-load the configuration, refreshing all parameters set in the config file [3]:

intelmq/intelmq/lib/bot.py

Lines 786 to 788 in 7674949

 for option, value in params.get('parameters', {}).items(): 
     setattr(self, option, value) 
     self.__log_configuration_parameter("runtime", option, value) 

Failing edge case

Requirements

• Assuming you want parser to detect the feed, so you didn't set any value in parametrs.feedname in the parser config.
• You have also installed our Debian package or manually configured log rotation with reloading.
• Your IntelMQ instance is working long enough for log rotation to occur.

Case steps

1. After starting bot, the mode is correctly detect [1], and then handling every report a correct feed is detected and set [2].
2. The log rotation occurs, and the parser is reloaded.
3. The configuration is refreshed, overriding all fields set in the config file [3].
4. Bot init method is called once more and tries to detect mode, but self.feedname is still set with the last value before reload, and the fixed mode is detected [0].
5. On every next report handled, the last previously detected feed is used

Workarounds

Set the feedname as null in the config. However, due to the IntelMQ Manager issue with nulls certtools/intelmq-manager#294 this may be changed to an empty string and case the fail again.

Fixes

• The quickest - clean the feedname in bot's shutdown method. If the config has a value, it will be set.

However, this issue can happen to any bot relying on default values and modifying their config. Probably it should be later solved on the IntelMQ library level.

[elsif2]

Fix added to shadowserver-dynamic-config.

[kamil-certat]

Could you cherrypick it to the main develop branch? I would like to push releasing patch soon.