Oneapi with moonshot not work fine
xutengfei opened this issue · comments
Self Checks
- This is only for bug report, if you would like to ask a question, please head to Discussions.
- I have searched for existing issues search for existing issues, including closed ones.
- I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
- 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
- Please do not modify this template :) and fill in all the required fields.
Dify version
0.6.11
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
I'm useing OneAPI to limit ai request frequency. when i using moonshot directly, function call with dify tool work perfectly.
but when using OneApi with Moonshot, it didn't work. I'm sure the option with "function call" in Oneapi with Moonshot is open!
✔️ Expected Behavior
using moonshot directly or using oneapi with moonshot should work same
❌ Actual Behavior
only using moonshot directly work fine
The issue with using OneAPI with Moonshot not working, while using Moonshot directly works fine in Dify version 0.6.11, is likely due to network accessibility problems. Specifically, if Dify and Moonshot are deployed using Docker, the localhost
address usually refers to the container itself, not the host machine or other containers. This can cause connection issues.
To resolve this, you need to expose the Moonshot service to the network. Instead of using localhost
, you should use host.docker.internal
to refer to the Docker host machine.
http://host.docker.internal:11434
This change should allow OneAPI to access the Moonshot service correctly [1].
The issue with using OneAPI with Moonshot not working, while using Moonshot directly works fine in Dify version 0.6.11, is likely due to network accessibility problems. Specifically, if Dify and Moonshot are deployed using Docker, the
localhost
address usually refers to the container itself, not the host machine or other containers. This can cause connection issues.To resolve this, you need to expose the Moonshot service to the network. Instead of using
localhost
, you should usehost.docker.internal
to refer to the Docker host machine.http://host.docker.internal:11434This change should allow OneAPI to access the Moonshot service correctly [1].
thanks for your reply. I'm sure Oneapi with moonshot is set correctly! when i use that just for chatting,it works fine. but when i use for dify tool ( like function call?),it not work!
To resolve the issue where using OneAPI with Moonshot for function calls in Dify version 0.6.11 does not work correctly, you need to ensure that the function_calling_type
in the credentials is set to 'tool_call'
. This is necessary because the get_customizable_model_schema
method checks this credential to determine if the model supports tool calls.
Here is the relevant part of the code:
def get_customizable_model_schema(self, model: str, credentials: dict) -> AIModelEntity | None:
return AIModelEntity(
model=model,
label=I18nObject(en_US=model, zh_Hans=model),
model_type=ModelType.LLM,
features=[ModelFeature.TOOL_CALL, ModelFeature.MULTI_TOOL_CALL, ModelFeature.STREAM_TOOL_CALL]
if credentials.get('function_calling_type') == 'tool_call'
else [],
fetch_from=FetchFrom.CUSTOMIZABLE_MODEL,
model_properties={
ModelPropertyKey.CONTEXT_SIZE: int(credentials.get('context_size', 4096)),
ModelPropertyKey.MODE: LLMMode.CHAT.value,
},
parameter_rules=[
ParameterRule(
name='temperature',
use_template='temperature',
label=I18nObject(en_US='Temperature', zh_Hans='温度'),
type=ParameterType.FLOAT,
),
ParameterRule(
name='max_tokens',
use_template='max_tokens',
default=512,
min=1,
max=int(credentials.get('max_tokens', 4096)),
label=I18nObject(en_US='Max Tokens', zh_Hans='最大标记'),
type=ParameterType.INT,
),
ParameterRule(
name='top_p',
use_template='top_p',
label=I18nObject(en_US='Top P', zh_Hans='Top P'),
type=ParameterType.FLOAT,
),
]
)
Ensure that your credentials dictionary includes 'function_calling_type': 'tool_call'
when invoking the model. Here is an example of how you might set up the credentials:
credentials = {
'api_key': 'your_api_key',
'function_calling_type': 'tool_call',
'context_size': 4096,
'max_tokens': 512,
# other necessary credentials
}
By setting the function_calling_type
to 'tool_call'
, the model will recognize that it supports tool calls, and the function calls should work correctly [1].
works perfect! Thanks, dify team!
The "Invalid request: agent message content is empty" error suggests that the prompt_messages
parameter might be empty or not properly populated when using OneAPI with Moonshot. Ensure that the prompt_messages
list contains valid PromptMessage
objects before invoking the method.
Here is an example of how to set up the prompt_messages
:
from core.model_runtime.entities.message_entities import UserPromptMessage, AssistantPromptMessage
prompt_messages = [
UserPromptMessage(content="Your user prompt here"),
AssistantPromptMessage(content="Your assistant prompt here")
]
credentials = {
'api_key': 'your_api_key',
'function_calling_type': 'tool_call',
'context_size': 4096,
'max_tokens': 512,
# other necessary credentials
}
model_parameters = {
'temperature': 0.7,
'top_p': 0.9,
'max_tokens': 512
}
# Assuming `moonshot_model` is an instance of `MoonshotLargeLanguageModel`
result = moonshot_model._invoke(
model="moonshot-v1-8k",
credentials=credentials,
prompt_messages=prompt_messages,
model_parameters=model_parameters
)
Ensure that the prompt_messages
list is not empty and contains the necessary message content. This should help resolve the "Invalid request: agent message content is empty" error [1].
The "Invalid request: agent message content is empty" error suggests that the
prompt_messages
parameter might be empty or not properly populated when using OneAPI with Moonshot. Ensure that theprompt_messages
list contains validPromptMessage
objects before invoking the method.Here is an example of how to set up the
prompt_messages
:from core.model_runtime.entities.message_entities import UserPromptMessage, AssistantPromptMessage prompt_messages = [ UserPromptMessage(content="Your user prompt here"), AssistantPromptMessage(content="Your assistant prompt here") ] credentials = { 'api_key': 'your_api_key', 'function_calling_type': 'tool_call', 'context_size': 4096, 'max_tokens': 512, # other necessary credentials } model_parameters = { 'temperature': 0.7, 'top_p': 0.9, 'max_tokens': 512 } # Assuming `moonshot_model` is an instance of `MoonshotLargeLanguageModel` result = moonshot_model._invoke( model="moonshot-v1-8k", credentials=credentials, prompt_messages=prompt_messages, model_parameters=model_parameters )Ensure that the
prompt_messages
list is not empty and contains the necessary message content. This should help resolve the "Invalid request: agent message content is empty" error [1].
I'm using dify web chat window. Not using api. the response is from dify.
by the way, Toocall response is good from Logs.
dify should send tool response with context to oneapi(moonshot). But Oneapi response with 400(no params)
seem like interrupt by some error
I'v seen this issue is marked as [feat:model-runtime],what does that mean? should i wait for latest release?
The issue you're experiencing with the "Invalid request: agent message content is empty" error when using OneAPI with Moonshot in Dify version 0.6.11 is likely related to the prompt_messages
parameter not being properly populated or formatted.
Ensure that your prompt_messages
list contains valid PromptMessage
objects and that they are correctly converted to the required dictionary format. Here is a snippet from the code that converts PromptMessage
objects to the required dictionary format:
def _convert_prompt_message_to_dict(self, message: PromptMessage, credentials: Optional[dict] = None) -> dict:
if isinstance(message, UserPromptMessage):
message = cast(UserPromptMessage, message)
if isinstance(message.content, str):
message_dict = {"role": "user", "content": message.content}
else:
sub_messages = []
for message_content in message.content:
if message_content.type == PromptMessageContentType.TEXT:
message_content = cast(PromptMessageContent, message_content)
sub_message_dict = {
"type": "text",
"text": message_content.data
}
sub_messages.append(sub_message_dict)
elif message_content.type == PromptMessageContentType.IMAGE:
message_content = cast(ImagePromptMessageContent, message_content)
sub_message_dict = {
"type": "image_url",
"image_url": {
"url": message_content.data,
"detail": message_content.detail.value
}
}
sub_messages.append(sub_message_dict)
message_dict = {"role": "user", "content": sub_messages}
elif isinstance(message, AssistantPromptMessage):
message = cast(AssistantPromptMessage, message)
message_dict = {"role": "assistant", "content": message.content}
if message.tool_calls:
message_dict["tool_calls"] = []
for function_call in message.tool_calls:
message_dict["tool_calls"].append({
"id": function_call.id,
"type": function_call.type,
"function": {
"name": function_call.function.name,
"arguments": function_call.function.arguments
}
})
elif isinstance(message, ToolPromptMessage):
message = cast(ToolPromptMessage, message)
message_dict = {"role": "tool", "content": message.content, "tool_call_id": message.tool_call_id}
elif isinstance(message, SystemPromptMessage):
message = cast(SystemPromptMessage, message)
message_dict = {"role": "system", "content": message.content}
else:
raise ValueError(f"Got unknown type {message}")
if message.name:
message_dict["name"] = message.name
return message_dict
Ensure that your prompt_messages
are correctly formatted and populated before making the API call. The PromptMessage
class and its subclasses (e.g., UserPromptMessage
, AssistantPromptMessage
, SystemPromptMessage
, ToolPromptMessage
) should be properly instantiated with the required attributes [1][2].
Regarding the [feat:model-runtime] label, it indicates that the issue is related to the model runtime feature. There have been recent updates in the latest version (v0.6.12-fix1) that address issues related to prompt_messages
and the feat:model-runtime
feature. These updates ensure that the handling of prompt_messages
and the construction of parameters for the glm-4v
model are more robust [3][4].
If the issue persists, it might be related to the specific implementation of the OneAPI with Moonshot, and you may need to wait for the latest release to resolve this issue, especially since it is marked as [feat:model-runtime].
The issue you're experiencing with the "Invalid request: agent message content is empty" error when using OneAPI with Moonshot in Dify version 0.6.11 is likely related to the
prompt_messages
parameter not being properly populated or formatted.Ensure that your
prompt_messages
list contains validPromptMessage
objects and that they are correctly converted to the required dictionary format. Here is a snippet from the code that convertsPromptMessage
objects to the required dictionary format:def _convert_prompt_message_to_dict(self, message: PromptMessage, credentials: Optional[dict] = None) -> dict: if isinstance(message, UserPromptMessage): message = cast(UserPromptMessage, message) if isinstance(message.content, str): message_dict = {"role": "user", "content": message.content} else: sub_messages = [] for message_content in message.content: if message_content.type == PromptMessageContentType.TEXT: message_content = cast(PromptMessageContent, message_content) sub_message_dict = { "type": "text", "text": message_content.data } sub_messages.append(sub_message_dict) elif message_content.type == PromptMessageContentType.IMAGE: message_content = cast(ImagePromptMessageContent, message_content) sub_message_dict = { "type": "image_url", "image_url": { "url": message_content.data, "detail": message_content.detail.value } } sub_messages.append(sub_message_dict) message_dict = {"role": "user", "content": sub_messages} elif isinstance(message, AssistantPromptMessage): message = cast(AssistantPromptMessage, message) message_dict = {"role": "assistant", "content": message.content} if message.tool_calls: message_dict["tool_calls"] = [] for function_call in message.tool_calls: message_dict["tool_calls"].append({ "id": function_call.id, "type": function_call.type, "function": { "name": function_call.function.name, "arguments": function_call.function.arguments } }) elif isinstance(message, ToolPromptMessage): message = cast(ToolPromptMessage, message) message_dict = {"role": "tool", "content": message.content, "tool_call_id": message.tool_call_id} elif isinstance(message, SystemPromptMessage): message = cast(SystemPromptMessage, message) message_dict = {"role": "system", "content": message.content} else: raise ValueError(f"Got unknown type {message}") if message.name: message_dict["name"] = message.name return message_dictEnsure that your
prompt_messages
are correctly formatted and populated before making the API call. ThePromptMessage
class and its subclasses (e.g.,UserPromptMessage
,AssistantPromptMessage
,SystemPromptMessage
,ToolPromptMessage
) should be properly instantiated with the required attributes [1][2].Regarding the [feat:model-runtime] label, it indicates that the issue is related to the model runtime feature. There have been recent updates in the latest version (v0.6.12-fix1) that address issues related to
prompt_messages
and thefeat:model-runtime
feature. These updates ensure that the handling ofprompt_messages
and the construction of parameters for theglm-4v
model are more robust [3][4].If the issue persists, it might be related to the specific implementation of the OneAPI with Moonshot, and you may need to wait for the latest release to resolve this issue, especially since it is marked as [feat:model-runtime].
after upgrade to the latest version(v0.6.12-fix1),it gave another error "lnvalid reauest: tool call id is not found"
To resolve the "Invalid request: tool_call_id is not found" error after upgrading to Dify version 0.6.12-fix1, ensure that the tool_call_id
is correctly set in the ToolPromptMessage
objects within your prompt_messages
list. The tool_call_id
must match the id
of the tool call in the AssistantPromptMessage
.
Here is a relevant snippet from the code that shows how tool_call_id
is used:
def _convert_prompt_messages_to_message_and_chat_histories(self, prompt_messages: list[PromptMessage]) \
-> tuple[str, list[ChatMessage], list[ChatStreamRequestToolResultsItem]]:
chat_histories = []
latest_tool_call_n_outputs = []
for prompt_message in prompt_messages:
if prompt_message.role == PromptMessageRole.ASSISTANT:
prompt_message = cast(AssistantPromptMessage, prompt_message)
if prompt_message.tool_calls:
for tool_call in prompt_message.tool_calls:
latest_tool_call_n_outputs.append(ChatStreamRequestToolResultsItem(
call=ToolCall(
name=tool_call.function.name,
parameters=json.loads(tool_call.function.arguments)
),
outputs=[]
))
else:
cohere_prompt_message = self._convert_prompt_message_to_dict(prompt_message)
if cohere_prompt_message:
chat_histories.append(cohere_prompt_message)
elif prompt_message.role == PromptMessageRole.TOOL:
prompt_message = cast(ToolPromptMessage, prompt_message)
if latest_tool_call_n_outputs:
i = 0
for tool_call_n_outputs in latest_tool_call_n_outputs:
if tool_call_n_outputs.call.name == prompt_message.tool_call_id:
latest_tool_call_n_outputs[i] = ChatStreamRequestToolResultsItem(
call=ToolCall(
name=tool_call_n_outputs.call.name,
parameters=tool_call_n_outputs.call.parameters
),
outputs=[{
"result": prompt_message.content
}]
)
break
i += 1
else:
cohere_prompt_message = self._convert_prompt_message_to_dict(prompt_message)
if cohere_prompt_message:
chat_histories.append(cohere_prompt_message)
if latest_tool_call_n_outputs:
new_latest_tool_call_n_outputs = []
for tool_call_n_outputs in latest_tool_call_n_outputs:
if tool_call_n_outputs.outputs:
new_latest_tool_call_n_outputs.append(tool_call_n_outputs)
latest_tool_call_n_outputs = new_latest_tool_call_n_outputs
if len(chat_histories) > 0:
latest_message = chat_histories.pop()
message = latest_message.message
else:
raise ValueError('Prompt messages is empty')
return message, chat_histories, latest_tool_call_n_outputs
Additionally, ensure that the tool_call_id
is included in the parameters when invoking the model, as shown in the _chat_generate
method:
def _chat_generate(self, model: str, credentials: dict,
prompt_messages: list[PromptMessage], model_parameters: dict,
tools: Optional[list[PromptMessageTool]] = None, stop: Optional[list[str]] = None,
stream: bool = True, user: Optional[str] = None) -> Union[LLMResult, Generator]:
client = AzureOpenAI(**self._to_credential_kwargs(credentials))
response_format = model_parameters.get("response_format")
if response_format:
if response_format == "json_object":
response_format = {"type": "json_object"}
else:
response_format = {"type": "text"}
model_parameters["response_format"] = response_format
extra_model_kwargs = {}
if tools:
extra_model_kwargs['tools'] = [helper.dump_model(PromptMessageFunction(function=tool)) for tool in tools]
if stop:
extra_model_kwargs['stop'] = stop
if user:
extra_model_kwargs['user'] = user
messages = [self._convert_prompt_message_to_dict(m) for m in prompt_messages]
response = client.chat.completions.create(
messages=messages,
model=model,
stream=stream,
**model_parameters,
**extra_model_kwargs,
)
if stream:
return self._handle_chat_generate_stream_response(model, credentials, response, prompt_messages, tools)
return self._handle_chat_generate_response(model, credentials, response, prompt_messages, tools)
Ensure that each ToolPromptMessage
has a valid tool_call_id
that corresponds to an existing tool call in the AssistantPromptMessage
[1][2][3][4][5].