Is there anyway that I can use learning rate warm-up during the training ?

[Shamane Siri]

I am using this repo to:

1. Continual Pre-training
2. SFT
3. DPR

For stage 1, I want to use a learning rate warm-up.

[lewtun]

Hello @shamanez the scripts in this repo are mostly focused on chat models, so for continual pretraining I recommend checking out the transformers script: https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py

There you can specify e.g. warmup_ratio in TrainingArguments to get the warmup.