Q: root/package logger configuration

[James Addison]

I hope asking questions via github issues is considered acceptable - not sure how else to do so other than tweeting.

Project package layout:

proj
proj/__init__.py
proj/__main__.py
proj/app.py
proj/services
proj/services/__init__.py
proj/services/discovery.py

I've got a subclass of Worker called Application in proj/app.py. I've got a service DiscoveryService located in proj/services/discovery.py. if I configure DEBUG mode in Application, it doesn't apply to the logger in DiscoveryService.

I believe this is simply because of Python's cascading logging - meaning, I need to set the logger in proj/__init__.py to address this. A single localized spot for logger configuration is ideal, so with that in mind, should I override on_setup_root_logger(self, _logger, _loglevel) in my Worker-based Application subclass to 'change' the root module logger? Like this maybe:

# proj/__init__.py
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# proj/app.py
import mode

from . import logger as module_logger

class Application(mode.Worker):
    ...

    def on_setup_root_logger(self, _logger, _loglevel):
        # what to do here to 'copy' the `_logger` into `module_logger`?

Assuming I'm on the right path, filling in that "what to do here" bit is where I'm confused. I'm sure it's simple though. 🙄

[Ask Solem]

Thanks for spending time on this, I'm happy to help you get this set up!

Mode was split out from a project that we're going to open source soon, and it uses it extensively.
There's a lot of hooks and stuff in mode that lets you extend it, but hardly any of it is documented. Most of the time spent documenting anything is spent on this other project, but hope to document more of mode soon.

FIrst of all you should probably use mode.utils.logging.get_logger, as it performs the best practice of
adding a NullHandler:

# proj/__init__.py
from mode.utils.logging import get_logger

logger = get_logger(__name__)

setting the level you could do, but I'm not sure it's necessary.

You don't have to define on_setup_root_logger at all, as the Worker will set up the root logger for you based on the loglevel and logfile arguments.

What we do is that we have one logger for every module:

# `proj/services/discovery.py`
logger = get_logger(__name__)

These all propagate to the root logger, so the Worker only needs to configure that one.

Our project also has an App class, but it does not inherit from the worker.
In our case the "app" is something the user defines, kind of like in Celery:

my_app = Celery()

The app is a service that when started starts other services:

from mode import Service
from mode.utils.objects import cached_property

class App(Service):

    async def on_start(self):
       await self.add_runtime_dependency(self.consumer)
       await self.add_runtime_dependency(self.producer)

   @cached_property
   def consumer(self) -> Consumer:
        return Consumer(app=self)

    @cached_property
    def producer(self) -> Producer:
        Producer(app=self)

One specific issue with our App class is that users will define them at module scope.
Instantiating a service at module scope is bad, because it creates asyncio.Event's that require the event loop to be created.
To fix that we use mode.proxy.ServiceProxy to forward app.start() etc. to a separate class implementing the service:

from mode import Service
from mode.proxy import ServiceProxy

class App(ServiceProxy):

   @cached_property
   def _service(self):
       return AppService(app=self)

class AppService(Service):

    def __init__(self, app, **kwargs):
        self.app = app
        super().__init__(**kwargs)

    def on_start(self):
           await self.add_runtime_dependency(self.consumer)
           await self.add_runtime_dependency(self.producer)

If you don't create apps at module scope there is no reason to use ServiceProxy.

The worker we have is separate, and the app does not inherit from worker.

Instead the worker starts the app:

if __name__ == '__main__':
    Worker(Application(), loglevel='INFO', logfile=None).execute_from_commandline()

Using logfile=None means it will log to stderr.
Passing the logfile argument is all you need to setup the logging really. It will configure the root logger,
and any other logger will propagate to the root logger.

[James Addison]

Thanks for sharing detailed information, @ask! Now you've got me curious what the upcoming open-sourced project using mode will be...

I am adding a NullHandler() in that root logger, but had omitted it in my example above for brevity.

Sooo... it would appear that having logger.setLevel(logging.INFO) in my proj/__init__.py was the culprit. How embarrassing - that wall of text of mine above, all for nothing. mode did indeed set up the root logged without any other changes.

I'm curious however - are you suggesting that I should structure my project differently? You mention using ServiceProxy, having a non-Worker based Application class. At the moment, everything appears to be running fine, I'm not seeing anything asyncio.Event related that is wrong (errors, warnings).

Here's a bit more detail; my Application subclass of mode.Worker parses a YAML config file in order to determine which services to load and what configuration parameters to pass to the mode.Worker constructor via super(). So, you can see, the services are dynamically loaded.

# proj/app.py

class Application(mode.Worker):
    def __init__(self, *args, **kwargs):
        config_file = kwargs.pop('config_file', '') or 'config.yaml'
        try:
            with open(config_file, 'r') as f:
                config = yaml.load(f)
        except:
            raise Exception("'{}' config file not found or is invalid YAML".format(config_file))

        config = config.get('services') or {}
        enabled_services = config.pop('enabled', {})

        services = []
        for key, _cfg in enabled_services.items():
            cfg = config.copy()
            cfg.update(_cfg or {})

            if label == 'registry':
                services.append(RegistryService(config=cfg))
            elif ... # more services here

        verbose = kwargs.pop('verbose', False)
        if verbose:
            kwargs['loglevel'] = 'debug'
        else:
            kwargs['loglevel'] = 'info'

        super().__init__(*services, *args, **kwargs)


def run():
    parser = argparse.ArgumentParser()
    parser.add_argument("-c", "--config", dest="config_file", help="path to configuration file")
    parser.add_argument("-v", "--verbose", action="store_true", help="increase output verbosity")
    args = parser.parse_args()

    Application(config_file=args.config_file, verbose=args.verbose).execute_from_commandline()

# proj/__main__.py
if __name__ == '__main__':
    from .app import run
    run()

Because I have __main__.py above, I can start my app using:

python -m proj -c config.yaml

But I plan to set up a command 'shortcut' to run() via setup.py installation when i'm further along

I imagine the above layout can be improved upon, but I'm still moving from my old code to mode-based code. Feedback appreciated. Are there major changes I should do, based on a move to mode? Thanks again!

[James Addison]

@ask I'm loving the level of detail in your replies. Well explained, many thanks. That level of asyncio I think I had a tenuous grasp on before, but now it is much clearer.

Closing this issue... cheers, until the next one!