Giters

jekalmin / extended_openai_conversation

Home Assistant custom component of conversation agent. It uses OpenAI to control your devices.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Claude/Anthropic not working, support?

emanuelbaltaretu opened this issue · comments

commented

Hi,
For my use case the number of functions and instructions exceeds the max tokens for gpt3 so i am force to use gpt4 which is painfully slow and expensive and after a couple of days i gave up.

I was wondering if there is any change for supporting the latest Claude models that support up to 200k tokens that are also quite fast.

Not sure if it was supposed to work out of the box like the other self-hosted LLM but for me it did not work.

Here is the base url used: https://api.anthropic.com/v1

image

Any help or guidance would be greatly appreciated.

commented

Thanks for reporting an issue.

I just looked up API reference, but it would not be possible to use claude for following reasons:

  1. It uses its own python library anthropic rather than openai (not openai compatible).
  2. It seems it doesn't support function calling.
commented

Thanks for responding. Man you are the reason I am using home assistant.

So in the meantime yes I did find how about the different python library, but i also found this: https://github.com/jtsang4/claude-to-chatgpt
Also Calude does support function calling: https://docs.anthropic.com/claude/docs/functions-external-tools

I managed to fit my instructions into GPT-3 by rewriting everything. Merged some functions and made others multi purpose, exposed less entities and sadly also had to strip it's roleplay character i worked so hard on but GPT-3 was spitting straight up my examples instead of improvising even when I strictly told him not to and also, funny enough, even using them in the wrong contexts, just pasting them randomly.
I do miss GPT-4 dearly as it's replies were work of art and was such a pleasure each time it was saying anything.
Another annoying thing is that i need to specifically tell gpt-3 how to do each step beforehand otherwise it gets confused and I need to babysit it constantly even after giving it all the steps it gets confused while GPT-4 was doing more than asked without telling it how to...
On the downside gpt-4 was crazy expensive and extremely slow, was costing me about 2$ a day vs 0.20$ with gpt-3...

I was thinking Claude would be the best of both worlds, smart, fast, with good context length so it can take lots of custom instructions, and also cheap. Maybe some time in the future.

But that's out of scope for this repo for now, sorry for venting, I will close this issue now

commented

@jekalmin Hi, love your work! Would you be open to accepting a PR which would implement the Anthropic (and maybe Cohere or Mistral) API library?
I'm considering working on it as their new models seem like a good and cost-effective alternative. This page shows that tool use is available in beta right now so it should be possible to use it as a replacement.

I see though that the project is called OpenAI conversation, does that mean that support for other APIs is out of scope?

commented

I would personally love that, could maybe help if possible

commented

Since we got no response from the author I went a different route and made a simple proxy service which can translate requests made with OpenAI spec to Mistral, Cohere and Anthropic. If you set it up (using docker-compose its really easy), you can use claude models with Home Assistant. I'm currently using it with claude-3-haiku and its working pretty well.

From my testing Mistral models have issues with guessing when to use tools so these don't work well with HA and Cohere uses a different way of handling function variable which makes something like get_attributes in HA impossible using the current spec.

Claude if fine though so that's what I'm using, here's my project if you want to set it up: https://github.com/DawidPietrykowski/llm-converter