When to call setup_logging_queues()

[sla-te]

You stated: 'at some point in your app where it sets up logging' - Where or when exactly do I have to call it to make it work properly?

This iy my CustomLogger class which I instantiate in every new process anew, while passing a Queue to it, where I want it to log in.

class CustomLogger:
    __slots__ = ['name', 'log_queue', '_logger', 'no_gui']

    def __init__(self, name: str, queue: not None, create_output_file=False, output_folder=None, root_log_level: str = "info", no_gui=False, dev=False):
        if output_folder is None:
            output_folder = Path(__file__).parents[2]
        self.name = name
        self.log_queue = queue
        self.no_gui = no_gui
        dt_fmt = '%Y-%m-%d %H:%M:%S'

        if dev:
            clrd_fmt = f'%(log_color)s%(asctime)s.%(msecs)03d | %(levelname)-8s | {self.name}:%(funcName)s:%(lineno)s - %(message)s'
            non_clrd_fmt = f'%(asctime)s.%(msecs)03d | %(levelname)-8s | {self.name}:%(funcName)s:%(lineno)s - %(message)s'
        else:
            clrd_fmt = f'%(log_color)s%(asctime)s.%(msecs)03d | %(levelname)-8s | {self.name} - %(message)s'
            non_clrd_fmt = f'%(asctime)s.%(msecs)03d | %(levelname)-8s | {self.name} - %(message)s'

        if self.no_gui:
            import colorlog

            clrd_stream_handler = colorlog.StreamHandler()
            colored_formatter = colorlog.ColoredFormatter(
                clrd_fmt,
                datefmt=dt_fmt,
                reset=True,
                log_colors={
                    'DEBUG': 'fg_bold_cyan',
                    'INFO': 'fg_light_white',
                    'WARNING': 'fg_bold_yellow',
                    'ERROR': 'fg_bold_red',
                    'CRITICAL': 'fg_bold_purple',
                    'SUCCESS': 'fg_bold_green',
                    'HTTP_DEBUG': 'fg_light_white'
                },
                secondary_log_colors={},
                style='%'
            )
            clrd_stream_handler.setFormatter(colored_formatter)
            self._logger = colorlog.getLogger(name=self.name)
            self._logger.addHandler(clrd_stream_handler)
        else:
            self._logger = getLogger(name=self.name)
            non_clrd_stream_handler = StreamHandler()
            non_colored_formatter = Formatter(fmt=non_clrd_fmt, datefmt=dt_fmt)
            non_clrd_stream_handler.setFormatter(non_colored_formatter)
            self._logger.addHandler(non_clrd_stream_handler)

        self._logger.setLevel(self._set_root_log_level(root_log_level))
        queue_handler = QueueHandler(queue)
        self._logger.addHandler(queue_handler)

        if create_output_file and output_folder:

            _output_folder = Path(output_folder) / "logs"
            if not _output_folder.is_dir():
                _output_folder.mkdir(exist_ok=True)
            timestamp = datetime.now().strftime("%m-%d-%Y-%H")
            logfile = str((_output_folder / f"debug_{timestamp}.log").resolve())
            rotating_file_handler = ConcurrentRotatingFileHandler(logfile, "a", encoding='utf-8', maxBytes=5 * 1024 * 1024, backupCount=10)
            file_formatter = Formatter(fmt=f'%(asctime)s.%(msecs)03d | %(levelname)-8s - %(message)s', datefmt=dt_fmt)
            rotating_file_handler.setFormatter(file_formatter)
            self._logger.addHandler(rotating_file_handler)
        setup_logging_queues()  # <------ Call it here?

Is it correct to call it here? Wouldnt it create a new background Thread every time I instantiate the CustomLogger then? Do note I use multiple different queues I attach to logger.

[Preston Landers]

Yes, in general, you would call setup_logging_queues only one time per process, somewhere in your initialization / startup code after you have configured the rest of your logging handlers. You've got it in a class init here, which means it would get called every time this class is instantiated. Should be fine I guess as long as there's only one instance of this class per process. Or you could just pull it out of init and execute it later in your startup code.

That said, I think the real problem is that even if you do that (only call it once per process startup) it may still create excessive background threads. I think it's creating one thread per named logger when it should just create one total. It's been a while since I've looked at it but I've been meaning to take a closer look at that aspect.

Note that use of this background logging queue is in no way required to use Concurrent-Log-Handler. In theory it just moves some of the logging code execution to a background thread for performance and is totally optional. Due to the extra threads it may create (at least until it gets refactored) it may not help you much. Or you could just look at the source in concurrent_log_handler/queue.py just as a guide or starting point for your own implementation. It's not a long file. Thanks.

[sla-te]

Thank you for the thorough reply and clarification! Your package already fixed the "file in use error" without setup_logging_queues(), I just thought I could tweak a little further by calling it.