Support Anthropic Claude 3

Question

Support Anthropic Claude 3

enricoros opened this issue 8 months ago · comments

Enrico Ros commented 8 months ago

Looking for PR to add support for Claude 3, released today by Anthropic.

Jackson commented 8 months ago

#444

Bill Zhang · Answer 1 · Tue Mar 05 2024 06:17:03 GMT+0800 (China Standard Time)

This pr does not seem to work for me. Error message stating it requires the new messages api.

Jackson · Answer 2 · Tue Mar 05 2024 06:32:38 GMT+0800 (China Standard Time)

Ah there was an API change as well?

Bill Zhang · Answer 3 · Tue Mar 05 2024 09:44:06 GMT+0800 (China Standard Time)

It seems like it, other claude models work, just not Claude 3

Penagwin · Answer 4 · Wed Mar 06 2024 04:14:20 GMT+0800 (China Standard Time)

https://docs.anthropic.com/claude/docs/models-overview#model-comparison

All Claude 3 models only support the Messages API and I think currently only the completions API is implemented for Claude here?

Enrico Ros · Answer 5 · Wed Mar 06 2024 06:35:20 GMT+0800 (China Standard Time)

Correct @Penagwin - they just introduced the new Messages API and the Claude 3 models are the ones to only support that new API. We'll need to update the backend code.

Bill Zhang · Answer 6 · Thu Mar 07 2024 02:04:49 GMT+0800 (China Standard Time)

Is this the backend code for claude: https://github.com/enricoros/big-AGI/blob/main/src/modules/llms/server/anthropic/anthropic.router.ts

Might be interested in taking a crack at this

Michael Brown · Answer 7 · Thu Mar 07 2024 09:58:41 GMT+0800 (China Standard Time)

Hey everyone! 👋

I've taken a crack at adding support for Anthropic Claude 3 models, as mentioned by @enricoros and @Penagwin. The changes primarily focus on updating the backend code in anthropic.router.ts to use the new Anthropic Messages API.

Here's a quick rundown of what I've done:

🔧 Updated the backend code to use the new Messages API, as required by Claude 3 models.
🆕 Implemented support for Claude 3 models, so users can now leverage these powerful new tools.
📝 Updated the documentation and comments to reflect the changes and provide clarity on usage.
🧪 Added tests to ensure the Claude 3 functionality works as expected and to maintain code quality.
I've submitted a pull request with all the details, and I'm excited to hear your thoughts and feedback! Let's work together to get this merged and provide our users with access to the latest and greatest from Anthropic.

Thanks for your time, and let me know if you have any questions or suggestions! 😄

#450

Enrico Ros · Answer 8 · Thu Mar 07 2024 10:30:57 GMT+0800 (China Standard Time)

Thanks @slapglif - looking into it. Seems like there's no Streaming support; looking into that.

Michael Brown · Answer 9 · Thu Mar 07 2024 10:41:05 GMT+0800 (China Standard Time)

@enricoros
no problem, been using this software since the day i hit you up, least i can do - i replied to the pr:

@enricoros as to the fidelity of the fix, upon further inspection, i receive some cases with:
 ○ Compiling /api/trpc-node/[trpc] ...
 ✓ Compiled /api/trpc-node/[trpc] in 1037ms (700 modules)
Error in stream: [SyntaxError: Expected ',' or '}' after property value in JSON at position 39]
this is likley a parsing error, ill see to it shortly, as for the lack of streaming support - the current situation is that streaming is supported and streams to console, but i wanted to push a working version before attacking the streaming to next bug,

to address the formatting - sorry yeah python guy, can you define your standard for me so i can use an llm to reformat?

#450 (comment)