ashleve / lightning-hydra-template

PyTorch Lightning + Hydra. A very user-friendly template for ML experimentation. ⚡🔥⚡

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Fixed seeding before instantiation of mlflow logger causes identical run names

ChristianHinge opened this issue · comments

MLFlow generates random run names using the random module. Since random.seed is set before instantiation of the mlflow logger, the same run name is always generated.

https://stackoverflow.com/questions/76053273/how-can-i-set-the-seed-that-mlflow-uses-to-randomly-generate-run-names

My current fix is to instantiate the loggers before the line:
L.seed_everything(cfg.seed, workers=True)

and forcing the setup of the logger during instatiation by logging some dummy variable:

...
    for _, lg_conf in logger_cfg.items():
        if isinstance(lg_conf, DictConfig) and "_target_" in lg_conf:
            log.info(f"Instantiating logger <{lg_conf._target_}>")
            _logger = hydra.utils.instantiate(lg_conf)
            logger.append(_logger)
            _logger.log_metrics({"dummy": 0})

    return logger

If the above is not added, the setup of the logger (which includes generating a random name), only happens at log_hyperparameters, and thus the seed would already be set and the run name would again be identical.

This feels like a hacky solution. Do you have any suggestions on a propper way to address the problem?