InstructBLIP Pipeline

Table of Contents

📝 Overview

📝 Overview

This is a pipeline providing InstructBLIP multimodal operation for Vicuna family models running on oobabooga/text-generation-webui.

⏩ Just let me run the thing

Clone this repo into your extensions/multimodal/pipelines folder and run the server with --multimodal enabled and a preferred pipeline. Use AutoGPTQ to load.

> cd text-generation-webui
> cd extensions/multimodal/pipelines
> git clone https://github.com/kjerk/instructblip-pipeline
> cd ../../../
> python server.py --auto-devices --chat --listen --loader autogptq --multimodal-pipeline instructblip-7b

👀 Examples

|

✅ Recommended Settings

Generation Parameter Presets:

LLaMA-Precise
Big O

💸 Requirements

AutoGPTQ loader (ExLlama is not supported for multimodal)
No additional dependencies from textgen-webui

VRAM Requirements

instructblip-7b + vicuna-7b	~6GB VRAM
instructblip-13b + vicuna-13b	11GB VRAM

The vanilla Vicuna-7b + InstructBLIP just barely runs on a 24GB gpu using huggingface transformers directly, and the 13b at fp16 is too much, thanks to optimization efforts and Quantized models/AutoGPTQ, on textgen-webui with AutoGTPQ, InstructBLIP and Vicuna can comfortably run on 8GB to 12gb of VRAM. 🙌

Provided Pipelines

'instructblip-7b' for Vicuna-7b family
'instructblip-13b' for Vicuna-13b family

Tested Working Models

instructblip-7b
instructblip-13b
- vicuna-13b-v1.1-4bit-128g (Standard)
- vicuna-13b-v1.3-4bit-128g
- airoboros-gpt4-1.4-13b-4bit-128g
- nous-hermes-13b-4bit-128g
- Manticore-Chat-Pyg-Guanaco-4bit-128g
- gpt4-x-vicuna-13b-4bit-128g
- vicuna-13b-v0-4bit-128g (Outmoded)

Non-Working Models

wizard-vicuna-13b-4bit-128g

🖥️ Inference

Due to the already heavy VRAM requirements of the respective models, the vision encoder and projector are kept on CPU and are relatively quick, while the Qformer is moved to GPU for speed.