How to conduct model fine-tuning training and what is the required data magnitude?

[Alex Xie]

[XuDong Frank Wang]

Hi, you can follow our instructions on model training to finetune the model on your datasets. If you fine-tune the pretrained InstanceDiffusion model with LORA, the data magnitude can be significant reduced. The exact data magnitude depends on your task.

[Alex Xie]

Do you have any plans to open source for lora training?

[XuDong Frank Wang]

Hi, this repo is mainly for reproducing the results reported in our paper. Currently, we don't have plans to support additional new tasks or lora training. But you can check https://github.com/cloneofsimo/lora to learn how to add LORA to stable diffusion (the base model we used in this repo). Thanks!

[S.W.Peng]

Thanks you for your great job,Can I use your model to do simpler training on a RTX3090 24g?

[XuDong Frank Wang]

You may want to use flash attention (and or deepspeed) during the training time if you want to train the model using RTX 3090. Flash attention was implemented already, you can set it as True in the .yaml config file.

Hope it helps.

[XuDong Frank Wang]

Closing it for now, please reopen-it if you have more questions.