DrishtiShrrrma / deepspeed-huggingface

References

1. https://discord.com/channels/879548962464493619/1050420105613430794/1054800474349514753
2. https://discord.com/channels/879548962464493619/1052622386064793660/1053321191072464896
3. https://discord.com/channels/879548962464493619/1052622386064793660/1053323395107913738
4. https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event#deepspee
5. https://huggingface.co/docs/transformers/v4.19.2/en/main_classes/deepspeed#deployment-in-notebooks
6. https://discuss.huggingface.co/t/deepspeed-integration-with-trainer-in-colab-crashing-typeerror-object-init-takes-exactly-one-argument-the-instance-to-initialize/28255

Models to be Assessed

1. SpeechT5
2. Whisper
3. Wav2Vec2

Things to be Considered

1. Random Seed (https://discuss.huggingface.co/t/fixing-the-random-seed-in-the-trainer-does-not-produce-the-same-results-across-runs/3442)
2. Training Steps
3. train_batch_size

Constraints to be Assessed

1. Training Speed
2. Inference Speed
3. Does DeepSpeed affect Model Performance too?

Observations