prompt image alignment experiment LLaVA-server
jeeyung opened this issue · comments
Hello!
Could you please elaborate how you set up LLaVA-server? I am struggling with using the server.
Yeah LLaVA-server is a bit tricky to get working. You need to use this repository from Kevin Black. Is there something specific that you are unsure about?
The main thing is just to install the pyproject.toml. The other thing that I recommend is making this in a different conda environment since I believe that this requires a different version of transformers than RLCM does.
Oh I see that you had OOM -- I think it's worth trying 7b and seeing if you can get that to work. Also you could quantize the model more. The GPUs that I used had 48gb of memory (nvidia a6000s)
Hi, Thank you for the quick response.
Actually, I encountered many issues including OOM, out of index error for embedding matrix, no config for LLaVA etc...
Even though I made it running with several tricks from here and there, but it results in 0 bert score many times.
As one example of errors, didn't you encounter the issue below related to llavaconfig?
