zccheng77 / VChat-BigDL

🤖VChat-BigDL: A ChatBot for Video Understanding Based on BigDL

🤖 VChat-BigDL: A ChatBot for Video Understanding Based on BigDL

Powered by BigDL, Llama, Clip, Whisper, Tag2Text, Helsinki, LangChain and inspired by showlab/Vlog, we turn a video into a long document which records visual and audio information. Then we can chat over the record only using Intel CPU.

Pipeline

Demo

English	Chinese

Environment Preparing

System ---Windows

1. Create Conda Environment

conda  create -n vchat python=3.9 -y
activate vchat
pip install -r ./requirements.txt

2. Install FFmpeg

conda install -c conda-forge ffmpeg -y

3. Download Model Weight

Download Clip, Llama, Helsinki, all-MiniLM-L12-v2 by huggingface

python download_ckpt.py

Download Tag2Text weight from here

✅Please save weights to ./checkpoints.

4. Optimaize LLM

python LLM_low_bit_optimize.py

5. Run with Gradio

python main_gradio.py

Tutorial

You can find the tutorial of VChat here.

❗Attention❗

1. Gradio Warning

Could not create share link. Missing file: D:\anaconda3\envs\vchat\lib\site-packages\gradio\frpc_windows_amd64_v0.2.

Please check your internet connection. This can happen if your antivirus software blocks the download of this file. You can install manually by following these steps:

1. Download this file: https://cdn-media.huggingface.co/frpc-gradio-0.2/frpc_windows_amd64.exe.
2. Rename the downloaded file to: frpc_windows_amd64_v0.2.
3. Move the file to this location: .\anaconda3\envs\vchat\lib\site-packages\gradio.

2. One by One

Multiple people using the same demo at the same time will cause an error because the asynchronous running logic is not implemented.

Acknowledge

This project is based on BigDL, Vlog, Tag2Text, Whisper, Llama2, Helsinki, KTS, LangChain, Douyin_Tiktok_Scraper_PyPi.