After post_test_case_callbacks returns false, restart_callbacks is not called.

[tr4v3ler]

In my session:

post_test_case_callbacks=[check_alive]
restart_callbacks=[reset_target]

Implementation of check_alive:

def check_alive():
    g_target_ip_addr = "x.x.x.x"
    command = ["ping", "-c", "1", g_target_ip_addr]
    # noinspection PyTypeChecker
    message = "alive() sending a ping command to " + g_target_ip_addr
    print(message)
    try:
        subprocess.run(command, timeout=3)
    except subprocess.TimeoutExpired:
        return False
    else:
        print("PING success")
        return True

The following contents are mentioned in the boofuzz document:

restart_callbacks (list of method) – The registered method will be called after a failed post_test_case_callback Default None.

However, after check_alive returns false, reset_target is not called.

[Maximilian Lindner]

Sorry for the long delayed reply @tr4v3ler.

At first I thought this was a bug too. But then I remembered that we refactored the callback structure and implemented the CallbackMonitor.

The return values of the callbacks are discarded on purpose as documented here:

boofuzz/boofuzz/monitors/callback_monitor.py

Lines 50 to 62 in 70e5718

     def post_send(self, target=None, fuzz_data_logger=None, session=None): 
         """This method iterates over all supplied post send callbacks and executes them. 
         Their return values are discarded, exceptions are caught and logged: 
  
         - :class:`BoofuzzTargetConnectionReset <boofuzz.exception.BoofuzzTargetConnectionReset>` will log a failure 
         - :class:`BoofuzzTargetConnectionAborted <boofuzz.exception.BoofuzzTargetConnectionAborted>` will log an info 
         - :class:`BoofuzzTargetConnectionFailedError <boofuzz.exception.BoofuzzTargetConnectionFailedError>` will log a 
           failure 
         - :class:`BoofuzzSSLError <boofuzz.exception.BoofuzzSSLError>` will log either info or failure, depending on 
           if the session ignores SSL/TLS errors. 
         - every other exception is logged as an error. 
  
         All exceptions are discarded after handling. 

Instead of a binary return code you can use more specific exceptions to inform boofuzz about a failure/error in your callback.
For your example a BoofuzzTargetConnectionFailedError seems to suite quite well for a ping check.

However, I miss an except block for BoofuzzFailure logging a failure. I'm not 100% sure what the original intention behind this exception was, but from its description it's the first one I'd chose to mark a test case failed for non-connection-related errors.

boofuzz/boofuzz/exception.py

Lines 5 to 10 in 70e5718

 class BoofuzzFailure(Exception): 
     """Raises a failure in the current test case. 
  
     This is most important when failures in the target may not be noticed until the next test case. In such a situation, 
     it may be necessary to abort the test case before a fuzz message is even sent, as the fuzz message may be poorly 
     defined outside the context of a valid partial protocol exchange. 

Any reason why we don't/shouldn't handle this exception for pre/post send callbacks @mistressofjellyfish @jtpereyda?

[Katharina Bogad]

Yes, that is the way CallbackMonitor works. It's post_send callback always returns true (i.e. no failure detected), it just logs exceptions. I don't remember exactly why I implemented it this way (sorry... it's been some time), but since I even documented that there is a real chance there was a reason. I can't find any technical one, so my best guess is that it is this way now because it has been this way before:

boofuzz/boofuzz/sessions.py

Lines 1560 to 1584 in b8045e6

 def _post_send(self, target): 
     if len(self._post_test_case_methods) > 0: 
         try: 
             for f in self._post_test_case_methods: 
                 self._fuzz_data_logger.open_test_step('Post- test case callback: "{0}"'.format(f.__name__)) 
                 f(target=target, fuzz_data_logger=self._fuzz_data_logger, session=self, sock=target) 
         except exception.BoofuzzTargetConnectionReset: 
             self._fuzz_data_logger.log_fail(constants.ERR_CONN_RESET_FAIL) 
         except exception.BoofuzzTargetConnectionAborted as e: 
             self._fuzz_data_logger.log_info( 
                 constants.ERR_CONN_ABORTED.format(socket_errno=e.socket_errno, socket_errmsg=e.socket_errmsg) 
             ) 
         except exception.BoofuzzTargetConnectionFailedError: 
             self._fuzz_data_logger.log_fail(constants.ERR_CONN_FAILED) 
         except exception.BoofuzzSSLError as e: 
             if self._ignore_connection_ssl_errors: 
                 self._fuzz_data_logger.log_info(str(e)) 
             else: 
                 self._fuzz_data_logger.log_fail(str(e)) 
         except Exception: 
             self._fuzz_data_logger.log_error( 
                 constants.ERR_CALLBACK_FUNC.format(func_name="post_send") + traceback.format_exc() 
             ) 
         finally: 
             self._fuzz_data_logger.open_test_step("Cleaning up connections from callbacks") 

The documentation is at fault here though - restart_target callbacks passed via Session arguments will be invoked if a Monitor returns false on a post_send callback, (which the post_test_case callbacks get shoved into), but not if it's done with the current CallbackMonitor. But AFAICT that was the case before my refactoring, so why is it a problem now? The answer to this seems to be that the complete monitor rework switched to the boolean return values which were never specified for the functions. So, technically, CallbackMonitor violates the behavior that is guaranteed from a Monitor based on BaseMonitor.

Should we change this? I honestly don't know. I'd much rather remove the callbacks to avoid things like these; but that can't be done without breaking legacy code. Since I consider the CallbackMonitor to be legacy compatibility code, there is no way that we can retrofit the return value of the callback to anything. It would default to a false-y value and thus break everything. So we must keep the "throwing away the return value of the callback"-Part. I'd also rather not add exception handling that might break legacy code (although I find it unlikely that this happens), especially because exception handling is a computation heavy operation compared to a simple "return False". But there is an argument to be made that incorrectly handling BoofuzzFailure here is a bug in itself.

@tr4v3ler, I'd suggest to use a custom monitor like so:

from boofuzz.monitors import BaseMonitor

class MyCustomMonitor(BaseMonitor):
    def post_send(self, target, fuzz_data_logger, session):
        g_target_ip_addr = "x.x.x.x"
        command = ["ping", "-c", "1", g_target_ip_addr]
        # noinspection PyTypeChecker
        message = "alive() sending a ping command to " + g_target_ip_addr
        print(message)
        try:
            subprocess.run(command, timeout=3)
        except subprocess.TimeoutExpired:
            fuzz_data_logger.log_fail("Got timeout")
            return False
        else:
            fuzz_data_logger.log_info("PING success")
            return True

    def restart_target(self, target=None, fuzz_data_logger=None, session=None):
        # your code here

Keep in mind though that it might be better to implement start_target and stop_target (which will be called independently by the base class and can provide more granular information if i.e stopping worked, but the target doesn't start anymore).

@SR4ven @jtpereyda To address the bug, I'd suggest two things:

1. fixing the documentation regarding the wording of "failure"
2. marking these callback-parameter functions as deprecated here:

   boofuzz/boofuzz/sessions.py

   Lines 511 to 529 in 70e5718

    if pre_send_callbacks is None: 
        pre_send_methods = [] 
    else: 
        pre_send_methods = pre_send_callbacks 
     
    if post_test_case_callbacks is None: 
        post_test_case_methods = [] 
    else: 
        post_test_case_methods = post_test_case_callbacks 
     
    if post_start_target_callbacks is None: 
        post_start_target_methods = [] 
    else: 
        post_start_target_methods = post_start_target_callbacks 
     
    if restart_callbacks is None: 
        restart_methods = [] 
    else: 
        restart_methods = restart_callbacks 

   
   IMHO, no matter how many exceptions we add, we can never achieve the granularity of the real world. Logging the failure itself should be done by a Monitor implementation, not by some wrapper guessing the failure that happened without any ability to properly log what has happened in the first place.