Feature Request: oobabooga/text-generation-webui as backend support
TheElo opened this issue · comments
Hey, interesting tool you have there, would it be possilbe to support also this API backend for local language models? Thank you in advance!
Hi, thank you!
Can you please explain more about what that support should look like? How do you want to use PySpur and WebUI together?
Hey, I meant to use that as a backend, similiar how you use Ollama as a backend to serve models for inference.
Currently the choice on your (pyspur) side is very limited to only a small number of models. Ollama for example supports a lot more models but instead of pulling the list of models ollama has to pyspur so the user can select which model to use (see Open-WebUI), you provide a list, forcing the user to select the models you want them to select.
I had issues setting up ollama properly so I hoped to get another backend, which is even more open about using diffrent models than your selection and even more huzzle free than ollama (it renames models and stores them in a weird way, obaa leaves things as is)
Potentially unrelated rant that should be 3 feature request:
By now I gave up on pyspur, will give it another spin in a year or so, setting up n8n locally just worked better:
I failed to setup Ollama as the server, running local, running in docker, connecting networks, nothing really allowed pyspur to access ollama, tried bunch of link variation - which is very annoying as the UI doesn't handle it but you have to use a .env file and rebuild the docker container every time.
[1. Allow Ollama configuration in PySpur UI, while it runs]
[2. Give better feedback to the user why the connection to ollama failed, same settings as for n8n do not work, and due to lack of UI, it's time consuming to experiment - "Workflow Failed" is not helpful, why did it fail?]
Even if it would connect pyspur it does not allow to select available models, but only ones from an oddly spefic weird list: like it support the tiny LLama3.2 models, but not the bigger better ones LLama 3.1 models, and as it's using the default tag, it will pull the heavily quantized model (Q4_K_M, instead of Q8 or at least Q6). As I'm debugly blind I thought maybe PySpur is requesting a model that Ollama does not have, so I downloaded the tiny llama3.2 with the same tag as your selection and I got the same error.
[3. Pull list of available models from ollama instead of providing a outdated/incomplete pre selection of models]
Thank you so much for the detailed feedback. This is very useful for us, and we will address these issues soon.
I'm sorry to hear that you had struggles with setting up Ollama. I'm aware of some users using PySpur + Ollama without issues, so if you are still interested in trying PySpur, please drop me a message, and I'm happy to assist you with setting it up either via email or on a call: jean@pyspur.dev