alibaba / conv-llava

ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models

Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao

Jun Song, Shiji Song, Gao Huang, Bo Zheng

[ English | 中文 ]

Abstract

High-resolution Large Multimodel Models (LMM) encounter the challenges of excessive visual tokens and quadratic visual complexity. Current high-resolution LMMs address the quadratic complexity while still generating excessive visual tokens. However, the redundancy in visual tokens is the key problem as it leads to more substantial compute. To mitigate this, we propose ConvLLaVA, which employs ConvNeXt, a hierarchical backbone, as the visual encoder of LMM to replace Vision Transformer (ViT). ConvLLaVA compresses high-resolution images into information-rich visual features, effectively avoiding the generation of excessive visual tokens. To enhance the capabilities of ConvLLaVA, we propose two critical optimizations.

• Since the low-resolution pretrained ConvNeXt underperforms when directly applied on high resolution, we update it to merge the gap.
• Furthermore, since ConvNeXt's original compression ratio is insufficient for much higher resolution inputs, we train a successive stage to further compress the visual tokens, effectively reducing redundancy.

These optimizations enable ConvLLaVA to support inputs of 1536x1536 resolution while generating only 576 visual tokens, accommodating images of arbitrary aspect ratios. Experimental results demonstrate that our method achieves competitive performance with state-of-the-art models on mainstream benchmarks.

Comparison between LLaVA and ConvLLaVA.

Release

• 2024/05/25: Checkpoints are released.
• 2024/04/17: Our code is released.

If you are interested in Large Multimodal Models or you have great ideas, please feel free to email with me: Chunjiang Ge.

Usage and License Notices: This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses, including but not limited to the OpenAI Terms of Use for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (e.g. Llama community license for LLaMA-2 and Vicuna-v1.5). This project does not impose any additional constraints beyond those stipulated in the original licenses. Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.

Contents

• Abstract
• Release
• Contents
• TODO
• Install
• Model Zoo
• Dataset
• Train
• Evaluation
• Citation
• Acknowledgement

TODO

• Add LMMs-eval supports.
• Add VLMEvalKit supports.
• Add xtuner supports.
• Release weights.
• Release inference code.

Install

1. Clone this repository and navigate to ConvLLaVA folder

git clone https://github.com/alibaba/conv-llava
cd conv-llava

1. Install Package

conda create -n convllava python=3.11 -y
conda activate convllava
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

3. Install additional packages for training cases

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Model Zoo

The performance on mainstream benchmarks are shown below:

Method	Resolution	Visual Tokens	LLM	MME	MMB	SEED	RealWorldQA	MMMU	MMVet	Text	Doc	POPE
ConvLLaVA	768	144	7B	1541	68	68.8	55.9	36.3	44.8	59.1	44.8	87.3
ConvLLaVA	1024	256	7B	1553	68.8	69.3	58.8	35.1	44.4	62.5	48.5	87.7
ConvLLaVA	1536	576	7B	1575	68.7	70.2	59.9	35.8	45.9	65.8	59	87.3

Method	Resolution	Visual Tokens	LLM	RefCOCO			RefCOCO+			RefCOCOg		Avg
				val	test-A	test-B	val	test-A	test-B	val	test
ConvLLaVA	768	144	7B	84.5	89.0	79.2	77.7	84.9	69.7	79.8	79.7	80.6
ConvLLaVA	1024	256	7B	85.5	89.6	78.8	79.3	86.1	70.3	80.6	81.2	81.4
ConvLLaVA	1536	576	7B	86.5	90.6	80.5	80.0	86.8	71.5	82.0	82.4	82.3

Please check out our Model Zoo for all public ConvLLaVA checkpoints, and the instructions of how to use the weights.

Dataset

Data we use is introduded in Data.md.

Train

We use the following hyperparameters for training ConvLLaVA.

Hyperparameters	Stage 1	Stage 2	Stage 3
Learning Rate	3e-4	2e-5	2e-5
Batch Size	256	256	128
Epochs	1	1	1
Warmup Ratio	0.03	0.03	0.03
Weight Decay	0	0	0
Optimizer	AdamW	AdamW	AdamW

The training scripts are in the scripts:

• Projector Initialzation: stage1
• Vision Language Pretraining: stage2
• Instruction Tuning: stage3

Evaluation

We support VLMEVALKIT and lmms-eval to evaluate our model now. See Evaluation.md for more details.

Citation

If you find LLaVA useful for your research and applications, please cite using this BibTeX:

@misc{ge2024convllava,
    title={ConvLLaVA: Hierarchical Backbones as Visual
Encoder for Large Multimodal Models},
    author={Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, Jun Song, Shiji Song, Gao Huang, Bo Zheng},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
    year={2024}
    eprint={2045.15738},
}

Acknowledgement

• Vicuna: the codebase LLaVA built upon, and our base model Vicuna-13B that has the amazing language capabilities!
• LLaVA: the codebase we built upon.