About using multiple GPUs to do lisa fine-tuning

[orderer0001]

If there are multiple GPUs, using the lisa method is also a direct script./scripts/run_finetune_with_lisa.sh? Do I need to set multi-GPU parameters?

[rpan]

Thanks for your interest in LMFlow! Currently we are working on the full multi-GPU support for LISA. Model parallelism is not integrated yet. If you run the script directly, it is data parallelism and may require more memory than its single GPU version.

Please stay tuned for our latest update, thanks for your understanding 🙏

[orderer0001]

Can other training methods be configured with multiple GPUs? Do I need to set parameters manually?

[orderer0001]

When will Lisa training’s support for multiple GPUs be updated?

[rpan]

Can other training methods be configured with multiple GPUs? Do I need to set parameters manually?

Yes. You may use ./scripts/run_finetune.sh, that script support model parallelism by utilizing deepspeed zero3.