Q: Whats the best way of combining configuration from both properties files and code?

[Roy Sindre Norangshol]

I wonder what is the best way to combine configuration from both properties files and code.
Currently we configure standard stuff by logevents.properties and logevents-<profile>.properties which is bundled with the application. However, fetching of secrets is done via another system, so SlackObserver etc needs to be added by code.

I didn't find the "right way" to ensure both sources of configuration was loaded before any logging were done.

This is my current approach which I'm not very proud of (in Kotlin):

Register my own class which subclasses the DefaultLogEventConfigurator as suggested by the documentation and loaded via Service Loader (META-INF/services/org.logevents.LogEventConfigurator file containing fullyqualified.ApplicationLogEventsConfigurator ):

class ApplicationLogEventsConfigurator() : DefaultLogEventConfigurator() {
    companion object {
        private var config: Config = Config.get()
        private val logger = LoggerFactory.getLogger(ApplicationLogEventsConfigurator::class.java)
    }

    // Visible for tests
    internal constructor(configFromTest: Config) : this() {
        config = configFromTest
    }

    override fun configure(factory: LogEventFactory) {
        resetConfigurationFromFiles(factory) // Ensure whatever defined in properties is set right away.

        val logConfig = mutableMapOf<String, String>()
        logConfig.putAll(loadConfigurationProperties())

        val observers = buildList {
            add("console")
            add("file")

            if (config.hasSlackConfiguration()) {
                logConfig.putIfAbsent("observer.slack", "SlackLogEventObserver")
                logConfig.putIfAbsent("observer.slack.threshold", "WARN")
                logConfig.putIfAbsent("observer.slack.username", "config.slackUsername")
                logConfig.putIfAbsent("observer.slack.slackUrl", config.slackUrl)
                logConfig.putIfAbsent("observer.slack.channel", config.slackChannel)
                logConfig.putIfAbsent("observer.slack.iconEmoji", config.slackIconEmoji)
                logConfig.putIfAbsent(
                    "observer.slack.markers.${LogMarkers.WORKER.name}.throttle",
                    "PT5M PT10M PT30M"
                )
                add("slack")
                logger.info("Enabling SlackLogEventObserver")
            }

            if (config.hasHumioConfiguration()) {
                logConfig.putIfAbsent("observer.humio", "HumioLogEventObserver")
                logConfig.putIfAbsent("observer.humio.elasticsearchUrl", config.humioElasitcsearchUrl)
                logConfig.putIfAbsent(
                    "observer.humio.elasticsearchAuthorizationHeader",
                    config.humioAuthorization
                )
                logConfig.putIfAbsent("observer.humio.index", config.humioIndex)
                logConfig.putIfAbsent(
                    "observer.humio.formatter.properties.environment",
                    config.instanceName
                )
                logConfig.putIfAbsent("observer.humio.idleThreshold", "PT2S")
                logConfig.putIfAbsent("observer.humio.cooldownTime", "PT1S")
                logConfig.putIfAbsent("observer.humio.maximumWaitTime", "PT30S")
                logConfig.putIfAbsent("observer.humio.suppressMarkers", "PERSONAL_DATA")
                logConfig.putIfAbsent("observer.humio.formatter.excludedMdcKeys", "secret")
                add("humio")
                logger.info("Enabling HumioLogEventObserver")
            }

            if (config.hasLogEventsServletConfiguration()) {
                logConfig.putIfAbsent("observer.servlet", "WebLogEventObserver")
                logConfig.putIfAbsent("observer.servlet.openIdIssuer", "https://auth.dataporten.no")
                logConfig.putIfAbsent("observer.servlet.clientId", config.logeventServletClientId)
                logConfig.putIfAbsent("observer.servlet.clientSecret", config.logeventServletClientSecret)
                logConfig.putIfAbsent("observer.servlet.requiredClaim.aud", config.logeventServletClientId)
                logConfig.putIfAbsent(
                    "observer.servlet.requiredClaim.sub",
                    config.logeventServletRequiredClaimSub
                )
                logConfig.putIfAbsent("observer.servlet.source", "DatabaseLogEventObserver")
                logConfig.putIfAbsent("observer.servlet.source.jdbcUrl", config.jdbcUrl)
                logConfig.putIfAbsent("observer.servlet.source.jdbcUsername", config.dbUsername)
                logConfig.putIfAbsent("observer.servlet.source.jdbcPassword", config.dbPassword)
                logConfig.putIfAbsent(
                    "observer.servlet.source.logEventsTable",
                    config.logeventDatabaseTable
                )
                add("servlet")
                logger.info("Enabling WebLogEventObserver")
            }
        }
        logConfig.put("root", "INFO ${observers.joinToString()}")

        // Apply config over again, existing config from properties files and config from code.
        applyConfigurationProperties(factory, logConfig)
    }
}

Something tells me there is a nicer way to do this?

[Johannes Brodwall]

It's difficult to give a general answer since there are several possible scenarios with combination of configuration from different sources.

org.logevents.config.DefaultLogEventConfigurator has several methods that are suitable for overriding. Perhaps the best one is loadPropertiesFromFile():

@Override
    protected Map<String, String> loadPropertiesFromFiles(List<String> configurationFileNames) {
        Map<String, String> properties = super.loadPropertiesFromFiles(configurationFileNames);

        properties.putIfAbsent("observer.file.filename", "logs/%application.log");
        properties.putIfAbsent("observer.file.archivedFilename", "logs/%date{yyyy-MM}/%application-%date.log");
        properties.putIfAbsent("logevents.jmx", "true");
        properties.putIfAbsent("logevents.installExceptionHandler", "true");
        properties.putIfAbsent("observer.*.packageFilter", getPackageFilter());

        return properties;
    }

This is pretty similar to what you're doing. But you seem to have some other configuration system that you get properties from as well, so it will depend a lot on what you want to do with this.

There's also a method org.logevents.config.DefaultLogEventConfigurator#installDefaultObservers which can be useful. With this, you can install things like the WebLogEventObserver and then use more flexible configuration to decide whether it should be used.

I've also added support for specifying root.observer.servlet=DEBUG (for example) instead of joining together as you do with logConfig.put("root", "INFO ${observers.joinToString()}")

[Johannes Brodwall]

Thinking a bit more, I think I would recommend:

1. Put the default configuration for observers like observer.humio.cooldown and observer.slack.marker.WORKER.throttle in a logevents.properties file in your classpath
2. Put configuration of what to log and when in a logevents.properties file in your working directory or in environment variables. This is things like root.observer.slack=WARN (LOGEVENTS_ROOT_OBSERVER_SLACK=WARN)
3. Subclass DefaultLogEventObserverConfigurator and override installDefaultObservers to get the rest.

Alternatively, you can just access LogEventFactory.getInstance() in your main method (or wherever) and do LogEventFactory.getInstance().addRootObserver(new LevelThresholdObserver(WARN, new SlackLogEventObserver(config.slackUrl))). (Please not that the logging threshold on a specific logger will override the threshold of an observer, so you may want to do LogEventFactory.getInstance().setRootLevel(Level.DEBUG), for example)

[Johannes Brodwall]

Next answer....

public class CustomConfigurator extends DefaultLogEventConfigurator {

    private Config config = Config.getInstance();

    @Override
    protected void configureRootLogger(LogEventFactory factory, Map<String, String> properties, Map<String, String> environment) {
        super.configureRootLogger(factory, properties, environment);

        getSlackObserver(properties).ifPresent(factory::addRootObserver);
    }

    private Optional<LogEventObserver> getSlackObserver(Map<String, String> configuration) {
        if (config.isSlackDisabled()) {
            return Optional.empty();
        }
        configuration.put("observer.slack.slackUrl", config.getSlackUrl());
        SlackLogEventObserver observer = new SlackLogEventObserver(configuration, "observer.slack");
        observer.setThreshold(Level.WARN);
        return Optional.of(observer);
    }
}

1. Override configureGlobalObserversFromProperties instead of configure (you could also override installDefaultObservers if you want logevents.properties to add them to specific categories or applyConfigurationProperties if you want to do other things with the LogEventFactory, or even configureGlobalObserversFromProperties)
2. Instantiate the observers directory instead of making the configuration load them. Some observers let you set many properties without using key-value-pairs, too (like threshold in the example)

As mentioned before, you can also get the LogEventFactory.getInstance directly if you don't want to use Config as a singleton.

[Johannes Brodwall]

I've adjusted the implementation of DefaultLogEventConfigurator and added some more documentation in the JavaDoc of LogEventConfigurator: https://jhannes.github.io/logevents/apidocs/org/logevents/LogEventConfigurator.html