- [2024.05.28] 📃 Our paper is accesible at arXiv now!
- [2024.05.20] 🔥 Our RLAIF-V-Dataset is used for training MiniCPM-Llama3-V 2.5, which represents the first end-side GPT-4V level MLLM!
- [2024.05.20] We open-source the code, weights (7B, 12B) and data of RLAIF-V!
We introduce RLAIF-V, a novel framework that aligns MLLMs in a fully open-source paradigm for super GPT-4V trustworthiness. RLAIF-V maximally exploits the open-source feedback from two key perspectives, including high-quality feedback data and online feedback learning algorithm. Notable features of RLAIF-V include:
- 💪 Super GPT-4V Trustworthiness via Open-source Feedback. By learning from open-source AI feedback, RLAIF-V 12B achieves super GPT-4V trustworthiness in both generative and discriminative tasks.
- 🤝 High-quality Generalizable Feedback Data. The feedback data usesed by RLAIF-V effectively reduce the hallucination of different MLLMs.
- ⚡️ Efficient Feedback Learning with Iterative Alignment. RLAIF-V exihibts both better learning efficiency and higher performance compared with the non-iterative approach.
We present the RLAIF-V Dataset, which is an AI generated preference dataset covering diverse range of tasks and domains. This open-source multimodal preference datasets contains more than 30K high-quality comparison pairs.
- Clone this repository and navigate to RLAIF-V folder
git clone https://github.com/RLHF-V/RLAIF-V.git
cd RLAIF-V
- Install package
conda create -n rlaifv python=3.10 -y
conda activate rlaifv
pip install -e .
- Install required spaCy model
wget https://github.com/explosion/spacy-models/releases/download/en_core_web_trf-3.7.3/en_core_web_trf-3.7.3.tar.gz
pip install en_core_web_trf-3.7.3.tar.gz
Model | Description | Download |
---|---|---|
RLAIF-V 7B | The most trustworthy variant on LLaVA 1.5 | 🤗 |
RLAIF-V 12B | Based on OmniLMM-12B, achieving super GPT-4V trustworthiness. | 🤗 |
We provide a simple example to show how to use RLAIF-V.
from chat import RLAIFVChat, img2base64
chat_model = RLAIFVChat('openBMB/RLAIF-V-7B') # or 'openBMB/RLAIF-V-12B'
image_path="./examples/test.jpeg"
msgs = "Describe in detail the people in the picture."
inputs = {"image": image_path, "question": msgs}
answer = chat_model.chat(inputs)
print(answer)
You can also run this example by executing the following script:
python chat.py
Inputs and expected outputs of the example
Question:
Why did the car in the picture stop?
Expected outputs:
In the picture, a car stopped on the road due to the presence of a sheep on the roadway. The car likely stopped to allow the sheep to safely move out of the way or avoid any potential accidents with the animal. This situation highlights the importance of being cautious and attentive while driving, especially in areas where animals may roam near roads.
- Prepare data (Optional)
If you can access huggingface dataset, you can skip this step, we will automatically download the RLAIF-V Dataset.
If you already downloaded the dataset, you can replace 'openbmb/RLAIF-V-Dataset' to your dataset path here at Line 38.
- Start training
Run the following command to start training.
bash ./script/train/llava15_train.sh
- Prepare COCO2014 annotations
The evaluation of Object HalBench relies on the caption and segmentation annotations from the COCO2014 dataset. Please first download the COCO2014 dataset from the COCO dataset's official website.
mkdir coco2014
cd coco2014
wget http://images.cocodataset.org/annotations/annotations_trainval2014.zip
unzip annotations_trainval2014.zip
- Inference, evaluation, and summarization
Please replace {YOUR_OPENAI_API_KEY}
with a valid OpenAI api-key.
# cd RLAIF-V
bash ./script/eval_rlaif_objhal.sh ./RLAIF-V_weight ./results/RLAIF-V ./coco2014/annotations {YOUR_OPENAI_API_KEY}
- Prepare MMHal Data
Please download the MMHal evaluation data here, and save the file in eval/data
.
- Run the following script to generate for MMHal Bench:
# cd RLAIF-V
bash ./script/eval/eval_rlaifv_mmhal.sh ./RLAIF-V_weight ./results/RLAIF-V {YOUR_OPENAI_API_KEY}
Usage and License Notices: The data, code, and checkpoint are intended and licensed for research use only. They are also restricted to uses that follow the license agreement of LLaMA, Vicuna, and Chat GPT. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.
- RLHF-V: The codebase we built upon.
- LLaVA: The instruction model and labeler model of RLAIF-V-7B.
- MiniCPM-V: The instruction model and labeler model of RLAIF-V-12B.
If you find our model/code/data/paper helpful, please consider cite our papers 📝 and star us ⭐️!
@article{yu2023rlhf,
title={Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback},
author={Yu, Tianyu and Yao, Yuan and Zhang, Haoye and He, Taiwen and Han, Yifeng and Cui, Ganqu and Hu, Jinyi and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong and others},
journal={arXiv preprint arXiv:2312.00849},
year={2023}
}
@article{yu2024rlaifv,
title={RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness},
author={Yu, Tianyu and Zhang, Haoye and Yao, Yuan and Dang, Yunkai and Chen, Da and Lu, Xiaoman and Cui, Ganqu and He, Taiwen and Liu, Zhiyuan and Chua, Tat-Seng and Sun, Maosong},
journal={arXiv preprint arXiv:2405.17220},
year={2024},
}