anakin87 / llama2-haystack

Using Llama2 with Haystack, the NLP/LLM framework.

llama2-haystack

(Experimental) Using Llama2 with Haystack

The notebook contains my hacky experiments in which I try to load and use Llama2 with Haystack, the NLP/LLM framework.

It's nothing official or well refined, but perhaps it may be useful to other people experimenting.

Notebook summary

1. Installed Transformers from the main branch (and other libraries) 📚
2. Loaded Llama-2-13b-chat-hf on Colab using 4-bit quantizazion, thanks to the great material shared by Younes Belkada 🙌
3. Disabled Tensor Parallelism, which caused some issues 🛠️
4. Installed a minimal version of Haystack
5. Found a hacky way to load the model in Haystack's PromptNode
6. Had a llama-zing chat session, from 🎧🎶 David Guetta to Don Matteo ⛪📿 (an Italian TV series)!