ludwig-ai / ludwig

Low-code framework for building custom LLMs, neural networks, and other AI models

Home Page:http://ludwig.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Remove target_module LoRA mapping for Phi-2 model

arnavgarg1 opened this issue · comments

The current PEFT LoRA modules map to the wrong target modules for Phi-2 since the architecture was recently updated.

I created a PR on the PEFT repo to update this to new target modules: huggingface/peft#1375

For now, we will fall back to target_modules for Phi-2 that lead to good model performance.