Giters

unslothai / unsloth

Finetune Llama 3.1, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

Home Page:https://unsloth.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Deployment Question

JesusPaz opened this issue · comments

commented

Hi, thank you for creating this repo. It's really helpful, it was the software I needed. I also do open source and I know that people never say thank you.

My question is: I'm new on this, what is the best tool that you recommend for running this on production? I was using ollama, but im not sure if it's the best.

Also, might be nice to allow discussions in this repo, that's the best way to answer this kind of questions and not create an issue.

Thank you very much.

commented

Oh Ollama is cool! We just made an Ollama notebook (not released yet, so in beta mode!) https://colab.research.google.com/drive/1WZDi7APtQ9VsvOrQSSC5DDtxq159j8iZ?usp=sharing

I would normally advise people to deploy via vLLM - Ollama is also good!