Flask logging - Other libraries - FileHandler creates duplicate log entries

[signag]

Description

In my web server, based on Flask 3.0.0, (raspi-cam-srv), I want to log to a log file.
For implementation, I followed the Flask Logging documentation.

For app and my own modules, I have added a FileHandler with

    # Configure loggers
    logsPath = os.path.dirname(app.instance_path) + "/logs"
    os.makedirs(logsPath, exist_ok=True)
    logFile = logsPath + "/raspiCamSrv.log"
    Path(logFile).touch(exist_ok=True)
    filehandler = logging.FileHandler(logFile)
    filehandler.setFormatter(app.logger.handlers[0].formatter)
    for logger in(
        app.logger,
        logging.getLogger("werkzeug"),
        logging.getLogger("raspiCamSrv.camera_pi"),
    ):
        logger.addHandler(filehandler)
        logger.setLevel(logging.ERROR)

As a result, I see duplicate entries for each log entry in the log file (see attachments, below).
The log output on the console has single entries.

Note

I did not add the default_handler to the loggers as described in the Flask Logging documentation.
When doing this, I get duplicate log entries also from the StreamHandler to the console.

How to reproduce the bug

This is the __init__.py: init.py.txt
This is the log file: raspiCamSrv.log
This is the console output; console.log

Expected behavior

Every log entry should occur only once

Environment:

• Python version: 3.11.2
• Flask version: 3.0.0
• Wekzeug version: 3.0.1

[Adrian]

just FYI, flask run should never be used for production setups, it's ONLY meant for development.
so dealing with the werkzeug dev server logger logging stuff should be a non-issue for production usage...

[signag]

Thanks, I know

[signag]

I found the solution:
It is sufficient to add the FileHandler only to the app Logger:

    # Configure loggers
    logsPath = os.path.dirname(app.instance_path) + "/logs"
    os.makedirs(logsPath, exist_ok=True)
    logFile = logsPath + "/raspiCamSrv.log"
    Path(logFile).touch(exist_ok=True)
    filehandler = logging.FileHandler(logFile)
    filehandler.setFormatter(app.logger.handlers[0].formatter)
    for logger in(
        app.logger,
        logging.getLogger("werkzeug"),
        logging.getLogger("raspiCamSrv.camera_pi"),
    ):
        logger.setLevel(logging.ERROR)
    app.logger.addHandler(filehandler)

Obviously, by some Flask magic, the other loggers get the FileHandler added, too.

With this modification, all log entries occur just once in the log file.

[David Lord]

It's a mistake in the docs. Instead of app.logger in the loop, it should be logging.getLogger(app.name). The docs were trying to say that logging should be configured before accessing app.logger, which is undermined if it's accessed during logging configuration. 🤦🏼‍♂️

[signag]

I tried with

    # Configure loggers
    logsPath = os.path.dirname(app.instance_path) + "/logs"
    os.makedirs(logsPath, exist_ok=True)
    logFile = logsPath + "/raspiCamSrv.log"
    Path(logFile).touch(exist_ok=True)
    filehandler = logging.FileHandler(logFile)
    filehandler.setFormatter(app.logger.handlers[0].formatter)
    for logger in(
        logging.getLogger(app.name),
        logging.getLogger("werkzeug"),
        logging.getLogger("raspiCamSrv.camera_pi"),
    ):
        logger.addHandler(filehandler)
        logger.setLevel(logging.ERROR)

... and get duplicate log entries in the log file again.

[David Lord]

You accessed app.logger a few lines up.

[David Lord]

From the docs you linked:

When you want to configure logging for your project, you should do it as soon as possible when the program starts. If app.logger is accessed before logging is configured, it will add a default handler.

[signag]

You accessed app.logger a few lines up.

OK, I had overseen this one.
But nevertheless, having the default_handler added, is OK for me.

Now, I tried with (this is the complete start of __init__.py and there is no access to a logger afterwards)

import os
from pathlib import Path
from flask import Flask
import logging
from flask.logging import default_handler
from picamera2 import Picamera2
import json

def create_app(test_config=None):
    # create and configure the app
    app = Flask(__name__, instance_relative_config=True)
    app.config.from_mapping(
        SECRET_KEY="dev",
        DATABASE=os.path.join(app.instance_path, "raspiCamSrv.sqlite"),
    )

    # ensure the instance folder exists
    try:
        os.makedirs(app.instance_path)
    except OSError:
        pass
    
    # Configure loggers
    logsPath = os.path.dirname(app.instance_path) + "/logs"
    os.makedirs(logsPath, exist_ok=True)
    logFile = logsPath + "/raspiCamSrv.log"
    Path(logFile).touch(exist_ok=True)
    filehandler = logging.FileHandler(logFile)
    filehandler.setFormatter(default_handler.formatter)
    for logger in(
        logging.getLogger(app.name),
        logging.getLogger("werkzeug"),
        logging.getLogger("raspiCamSrv.auth"),
        logging.getLogger("raspiCamSrv.auth_su"),
        logging.getLogger("raspiCamSrv.camCfg"),
        logging.getLogger("raspiCamSrv.camera_pi"),
        logging.getLogger("raspiCamSrv.config"),
        logging.getLogger("raspiCamSrv.home"),
        logging.getLogger("raspiCamSrv.images"),
        logging.getLogger("raspiCamSrv.info"),
        logging.getLogger("raspiCamSrv.settings"),
        logging.getLogger("raspiCamSrv.photoseries"),
        logging.getLogger("raspiCamSrv.photoseriesCfg"),
    ):
        logger.addHandler(default_handler)    
        logger.addHandler(filehandler)    
        logger.setLevel(logging.DEBUG)

... and get duplicate log entries for my own modules in the log file as well as in console output.
For module _internal, there is a single entry.

When I omit logger.addHandler(default_handler),
I get no log output on the console and again duplicate log output for my own modules in the log file

[David Lord]

Sounds like you have other issues. Logging configuration is confusing in general for Python. I'll fix the docs issue I commented about, but otherwise can't help with your specific issue here.

[signag]

Thanks a lot for your support.
I returned to the previous solution which works, even if it isn't according to the docs.