Bug: `model` must be defined despite docs saying that if not provided, it defaults to base url.
tomaarsen opened this issue · comments
Hello!
How to reproduce:
from easyllm.clients import huggingface
from easyllm.prompt_utils import build_llama2_prompt
huggingface.prompt_builder = build_llama2_prompt
# send a ChatCompletion request to count to 100
response = huggingface.ChatCompletion.create(
messages=[
{'role': 'user', 'content': 'Count to 100, with a comma between each number and no newlines. E.g., 1, 2, 3, ...'}
],
)Output:
Traceback (most recent call last):
File "...\demo.py", line 11, in <module>
response = huggingface.ChatCompletion.create(
File "...\easyllm\clients\huggingface.py", line 150, in create
client = InferenceClient(url, token=api_key)
UnboundLocalError: local variable 'url' referenced before assignment
Given that the docs say that the model "defaults to the base url if not provided", I would expect this to work.
Should I include a fix for this in #7? And if so, how should I tackle it? Set a default or require the model to be given?
- Tom Aarsen
Will look at this. The idea is that you can omit the model when you are using a different api_base method. That way you can use your own hosted version of TGI.
Understood, I'll leave this to you then.
#12 fixes it and i added an example as well with inference endpoints as reference: https://philschmid.github.io/easyllm/examples/inference-endpoints-example/#how-to-stream-chat-completion-requests