Function Calling with LM Studio Server Model

[Parker Settle]

Is there anyway to use this feature with chat-completion, instead of generator?

LM studio offers a server you can use your local models, in a chat-completion exactly like open-AI.

This is what the server script looks like:

from openai import OpenAI

# Example: reuse your existing OpenAI setup
from openai import OpenAI

# Point to the local server
client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed")

completion = client.chat.completions.create(
  model="local-model", # this field is currently unused
  messages=[
    {"role": "system", "content": "Always answer in rhymes."},
    {"role": "user", "content": "Introduce yourself."}
  ],
  temperature=0.7,
)

print(completion.choices[0].message)

It would be very nice for a plug and play in function calling.

[rizerphe]

The whole idea of this project is to restrict the generation token-by-token. Therefore, the short answer is no, it's not possible. This project is something you would integrate into a tool like LM Studio on the server side, to make it more suitable for function calling. Unless you want to change the server configuration, what you're trying to do just boils down to careful prompt engineering, and at most validation of the result. I have another project that does the validation and parsing with OpenAI. If you're receiving the full completion, though, there is no way to enforce the actually valid JSON being generated.

[Parker Settle]

The whole idea of this project is to restrict the generation token-by-token. Therefore, the short answer is no, it's not possible. This project is something you would integrate into a tool like LM Studio on the server side, to make it more suitable for function calling. Unless you want to change the server configuration, what you're trying to do just boils down to careful prompt engineering, and at most validation of the result. I have another project that does the validation and parsing with OpenAI. If you're receiving the full completion, though, there is no way to enforce the actually valid JSON being generated.

LM studio just added "{} Structured Output (enforce JSON)"; I am not really sure how to go about this. Do you have have any ideas? I am just trying to use function calling with the "tool params" that openAI uses so i do not have to convert my entire project when switching to a local LLM.