Giters

lhenault / simpleAI

An easy way to host your own AI API and expose alternative models, while being compatible with "open" AI clients.

Home Page:https://pypi.org/project/simple-ai-server/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Adding falcon gpt model - howto?

NicoJuicy opened this issue · comments

commented

How would i start with adding the falcon-40b and falcon-7b model?

Am i correct to just copy+rename eg. GPT-NeoXT-Chat-Base-20B

And then:

  • Adjust the 'get_models.py' to the relevant hugginface
  • Adjust the 'models.toml' file ( or create from template)

Would that work?

commented

Hi @NicoJuicy, thanks for the question.

I'd rather start with the MPT examples, they're a bit more advanced (transformers models seemed to have memory leaks on GPU and it's addressed in these examples). Otherwise yes that's the way to go.

You might also want to edit how chat input is formatted during preprocessing for chat models, and check if the device_map fits your needs.

commented

Thanks for the info. Some work has popped up and let's see if i can get back to this later on!

Nice work fyi!