Lightning-AI / pytorch-lightning

Pretrain, finetune and deploy AI models on multiple GPUs, TPUs with zero code changes.

Home Page:https://lightning.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

OnExceptionCheckpoint: training resumes if ckpt found, even if no ckpt_path provided

brijow opened this issue · comments

Bug description

However, if an OnExceptionCheckpoint is provided to the trainer's list of callbacks, then even if Trainer.fit is called without providing ckpt_path argument then the CheckpointConnector will search for a checkpoint in OnExceptionCheckpoint.dirpath, and if one is found it will be used to resume training.

Further, the warnings shown at the beginning of Trainer.fit when OnExceptionCheckpoint callback is enabled IMO are incorrect, (see logs below)

What version are you seeing the problem on?

v2.2

How to reproduce the bug

from lightning.pytorch.callbacks import OnExceptionCheckpoint, ModelCheckpoint
from lightning.pytorch import Trainer
from lightning.pytorch.demos.boring_classes import BoringModel

trainer = Trainer(
    # NOTE: either of these callback lists reproduce the issue, (and same warnings mentioned below)
    # callbacks=[OnExceptionCheckpoint("."), ModelCheckpoint(".", save_last=True)],
    callbacks=[OnExceptionCheckpoint("."), ModelCheckpoint(".", save_last=False)],
    max_epochs=3,
)
trainer.fit(model=BoringModel())

# calling `fit` again will result in resumed training from discovered checkpoint in cwd:
trainer.fit(model=BoringModel())

Error messages and logs

The following warnings are confusing and arguably wrong:

.../checkpoint_connector.py:126: `.fit(ckpt_path=None)` was called without a model. The last model of the previous `fit` call will be used. You can pass `fit(ckpt_path='best')` to use the best model or `fit(ckpt_path='last')` to use the last model. If you pass a value, this warning will be silenced.
.../checkpoint_connector.py:186: .fit(ckpt_path="last") is set, but there is no last checkpoint available. No checkpoint will be loaded. HINT: Set `ModelCheckpoint(..., save_last=True)`.```

### Environment

PyTorch Lightning Version: 2.2.1


### More info

_No response_