Giters

lhenault / simpleAI

An easy way to host your own AI API and expose alternative models, while being compatible with "open" AI clients.

Home Page:https://pypi.org/project/simple-ai-server/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[QUESTION] - Other use cases

talvasconcelos opened this issue · comments

commented

I'd like to run simpleAI in runpod for example, and expose an API endpoint, for roleplay chat (sometimes NSFW), so other's can use! I'd need to have a way of knowing which generation (inference) belongs to who. I believe a middleware of some sort could solve it, or some type of auth key that would go in the header, like open ai uses, right?

also, there's other option, wich is doing all that on a small fastapi app and make the requests from that to the simpleai endpoints (probably a better approach?).

can i use simple ai with a quantized model?

Sorry for the dumb questions here... tried to contact @lhenault on Twitter but no DM...

commented

Hey thanks for the message @talvasconcelos!

That's a cool project, feel free to share it by DM once you're ready (I have no issue with NSFW use cases). So about your questions:

  • You could indeed add a middleware (see here for an example for CORS, or here to define your own with FastAPI).
  • There is an id in the API response that you can probably use at some point, I think the best way to go is to manage all this through your app, logging what matters to you and simply calling the SimpleAI service.
  • SimpleAI by design can use any type of model, including quantized ones. You can perhaps have a look at the examples as a starter, it shouldn't be too hard to switch to a quantized one from there.

Thanks !

PS: regarding the contact thing, I suspect it's because you aren't following me, but my DM are open. :)

commented

Hi, yes I've lurked and saw all the examples for a while now... i found simpleAI while researching for this idea. I'm actually collaborating with the LNbits project, so I'm not a stranger to FastAPI. Altough backend is not my strongest skill, I think i can make something work!

My first idea was to allow users to be able to charge per call to their model api endpoint, and make it as an LNbits extension! But I may have to build a client for using with simpleAI! I'll keep you posted!