A conform map (as configuration) used for model names when exporting from autogroq
JaloWalt opened this issue · comments
Reason: I fail to find a model-api that works both in autogroq and in autogen.
The main problem with autogen (seems to be) is that groq-api limits the response-rate to a very low value. Remember you had to do special code also in autogroq?
So, when the export [button-press works nicely] from autogroq to autogen has been done then what do you do with the stuff now in autogen? You can not use the groq-models.
I went as far as using fireworks.ai api-inference (somewhat similar as groq-api) in autogen, but it seems a bit clumsy to need to go through all that was exported from autogroq and then change that (manually) in autogen to some model that (just/maybe/might) work with the autogroq generated stuff.
At this point also an awful heretical blasphemous idea is born: maybe the fireworks.api could be used in autogroq? So that both autogroq and autogen could use the same model.
Well, the meantime: any idea for the least-manual-effort in matching/making the models align in autogroq and autogen for actual immediate-continual-working after the autogroq-export-to-autogen? Maybe defining a default model in/for the export(ed) model to-be in autogen/crewai etc? Or something better: a conform map?
The fireworks.ai allure reason? It has a way higher response-rate limit(s). Actually in the very first contact the account-executive promised to me that the rate could be increased to 2,000 RPM if I so want.
In summary: If you have an/any example of an actual smooth continuing of the the work (in autogen studio) after exported from autogroq then that surely would be appreciated.
As a feature request: A conform map (as configuration) for model names when exporting from autogroq. That would at least resolve the model name differences between various apis.
Regards; Mr. Jalo Ahonen
P.S A list of few (fireworks.ai) models I have tried in autogen studio.
Structure in autogen; for example the first one:
baseurl is = https://api.fireworks.ai/inference/v1
model name is = accounts/fireworks/models/firefunction-v1
(Yeah, that's the [longish] model name(s). Not quite like in autogroq, even when using mixtral.)
$0/M Tokens
32,768 Max Context
https://api.fireworks.ai/inference/v1
accounts/fireworks/models/firefunction-v1
$0.9/M Tokens
200,000 Max Context
https://api.fireworks.ai/inference/v1
accounts/fireworks/models/yi-34b-200k-capybara
$0.2/M Tokens
32,768 Max Context
https://api.fireworks.ai/inference/v1
accounts/fireworks/models/openorca-7b
$0.9/M Tokens
65,536 Max Context
https://api.fireworks.ai/inference/v1
accounts/fireworks/models/mixtral-8x22b-instruct-hf
$0.9/M Tokens
65,536 Max Context
https://api.fireworks.ai/inference/v1
accounts/fireworks/models/mixtral-8x22b-instruct
$0.5/M Tokens
32,768 Max Context
https://api.fireworks.ai/inference/v1
accounts/fireworks/models/mixtral-8x7b-instruct-hf
$0.5/M Tokens
32,768 Max Context
https://api.fireworks.ai/inference/v1
accounts/fireworks/models/mixtral-8x7b-instruct
Correct on all counts. Groq's paid service will not suffer from this limitation.
In the meantime, you can click any gear icon and change the default model for all agents at once to a non-Groq option...
-jjg
Achieved (almost) what I wanted;
- Using the fireworks api in autogroq. Changed the file: AutoGroq/AutoGroq/config.py
[attached here as: config-py.txt]
config-py.txt
and added the provider file: AutoGroq/AutoGroq/llm_providers/fireworks_provider.py
[attached here as: fireworks_provider-py.txt]
fireworks_provider-py.txt
-
Tested that the two above files make fireworks-api to work in AutoGroq. OK, so the test prompt was simple: "Why do dogs bark at cats?" but it still/anyway verifies the basic functionality.
-
Exported to Autogen. Yes, it did do it.
-
In autogen; PROBLEMS
Running (in Autogen Playground) the [AutoGroq Workflow] which AutoGroQ exported to Autogen, will produce error.
Reason is following:
-
In AutoGroq to run correctly the base url must be: https://api.fireworks.ai/inference/v1/chat/completions
-
In Autogen to run correctly the base url must be: https://api.fireworks.ai/inference/v1
The official (and thus correct way) by fireworks.ai is: https://api.fireworks.ai/inference/v1/chat/completions
In conclusion: TODO
A. I will open a bug/error in/for autogen staff to handle.
B. I will use the specific config_local.py described in https://youtu.be/MGEXFuK7RDY (at 1:43+) to fix my (local) config to be resilient regardless AutoGroq updates.
C. Mr J.Gravelle will include the file "fireworks_provider.py" into AutoGroq build.
Make sure you share this on Discord, please... -jjg
I mean, you can do openrouter.ai as well:
https://openrouter.ai/docs/models
This sort of change should be in the new local config, not in config.py. (making config.py use more custom options like new providers or tweaking existing ones makes sense )
Achieved (almost) what I wanted;
- Using the fireworks api in autogroq. Changed the file: AutoGroq/AutoGroq/config.py
[attached here as: config-py.txt]
config-py.txtand added the provider file: AutoGroq/AutoGroq/llm_providers/fireworks_provider.py [attached here as: fireworks_provider-py.txt] fireworks_provider-py.txt
- Tested that the two above files make fireworks-api to work in AutoGroq. OK, so the test prompt was simple: "Why do dogs bark at cats?" but it still/anyway verifies the basic functionality.
- Exported to Autogen. Yes, it did do it.
- In autogen; PROBLEMS
Running (in Autogen Playground) the [AutoGroq Workflow] which AutoGroQ exported to Autogen, will produce error.
Reason is following:
- In AutoGroq to run correctly the base url must be: https://api.fireworks.ai/inference/v1/chat/completions
- In Autogen to run correctly the base url must be: https://api.fireworks.ai/inference/v1
The official (and thus correct way) by fireworks.ai is: https://api.fireworks.ai/inference/v1/chat/completions
In conclusion: TODO
A. I will open a bug/error in/for autogen staff to handle.
B. I will use the specific config_local.py described in https://youtu.be/MGEXFuK7RDY (at 1:43+) to fix my (local) config to be resilient regardless AutoGroq updates.
C. Mr J.Gravelle will include the file "fireworks_provider.py" into AutoGroq build.
As regarding the: TODO A. I created the following bug-report to autogen studio;
[Bug]: Autogen Studio App does not use the given baseurl #2903
microsoft/autogen#2903
I thought I did 'C.' already...
I thought I did 'C.' already...
Yes, you did. So TODO of C is done.
I will recheck the TODO of A with the newest Autogen Studio. (=0.0.56rc12 of May30th) [How to get it? See: https://github.com//issues/37]
The TODO of B is of course specific to each user of AutoGroq. I will let you know what I use once I have figured it out myself.
As to: TODO of A.
I updated the autogen-bug-item. BUG still remains.
I have a growing suspicion that autogen will continue making life difficult for everyone else and will (continue) to choose to pick OpenAI-type api-inferences base-urls only if they are given as ending by: "v1"
This means that AutoGroq might need to change any OpenAI-type [NOTE: the TYPE ones and not just actual OpenAI-ones] base-urls to end to "v1" when exporting to autogen. Either that or wait [during the wait having functionality breaking exports] the 2-months release-cycle when MAYBE the autogen has/will? fix this.
Agreed. We should wait for both Autogen to finish their re-write and AutoGroq to finish ours...