Custom batch selection for logging
bhosalems opened this issue · comments
Description & Motivation
Need to be able to select the same batch in every logging cycle. For generation pipelines similar to stable diffusion it is very hard to gauge the performance over training if we continue to choose random batches.
Pitch
User should have selective ability to choose the batch to log which will be constant for all the logging cycles.
Alternatives
Its possible to load the data again in train_btach_end() or validation_batch_end(), and call logging.
Additional context
No response
cc @Borda
I agree with your point...
So you want that user provides a list of indices from the dataset... And a batch is prepared at the end of epoch with elements corresponding to those indices... And logging them after forwarding them through model... Is it?
I think it would be better to log them at end of validation step ....
Pls assign this issue to me...
I would like to work on this issue.
You can open a pull request once you are done with the required code.