Oneapi with moonshot not work fine
xutengfei opened this issue · comments
Self Checks
- This is only for bug report, if you would like to ask a question, please head to Discussions.
- I have searched for existing issues search for existing issues, including closed ones.
- I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
- 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
- Please do not modify this template :) and fill in all the required fields.
Dify version
0.6.11
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
I'm useing OneAPI to limit ai request frequency. when i using moonshot directly, function call with dify tool work perfectly.
but when using OneApi with Moonshot， it didn't work. I'm sure the option with "function call" in Oneapi with Moonshot is open!
✔️ Expected Behavior
using moonshot directly or using oneapi with moonshot should work same
❌ Actual Behavior
only using moonshot directly work fine
The issue with using OneAPI with Moonshot not working, while using Moonshot directly works fine in Dify version 0.6.11, is likely due to network accessibility problems. Specifically, if Dify and Moonshot are deployed using Docker, the localhost address usually refers to the container itself, not the host machine or other containers. This can cause connection issues.
To resolve this, you need to expose the Moonshot service to the network. Instead of using localhost, you should use host.docker.internal to refer to the Docker host machine.
http://host.docker.internal:11434This change should allow OneAPI to access the Moonshot service correctly [1].
The issue with using OneAPI with Moonshot not working, while using Moonshot directly works fine in Dify version 0.6.11, is likely due to network accessibility problems. Specifically, if Dify and Moonshot are deployed using Docker, the
localhostaddress usually refers to the container itself, not the host machine or other containers. This can cause connection issues.To resolve this, you need to expose the Moonshot service to the network. Instead of using
localhost, you should usehost.docker.internalto refer to the Docker host machine.http://host.docker.internal:11434This change should allow OneAPI to access the Moonshot service correctly [1].
thanks for your reply. I'm sure Oneapi with moonshot is set correctly! when i use that just for chatting,it works fine. but when i use for dify tool ( like function call?),it not work!
To resolve the issue where using OneAPI with Moonshot for function calls in Dify version 0.6.11 does not work correctly, you need to ensure that the function_calling_type in the credentials is set to 'tool_call'. This is necessary because the get_customizable_model_schema method checks this credential to determine if the model supports tool calls.
Here is the relevant part of the code:
def get_customizable_model_schema(self, model: str, credentials: dict) -> AIModelEntity | None:
return AIModelEntity(
model=model,
label=I18nObject(en_US=model, zh_Hans=model),
model_type=ModelType.LLM,
features=[ModelFeature.TOOL_CALL, ModelFeature.MULTI_TOOL_CALL, ModelFeature.STREAM_TOOL_CALL]
if credentials.get('function_calling_type') == 'tool_call'
else [],
fetch_from=FetchFrom.CUSTOMIZABLE_MODEL,
model_properties={
ModelPropertyKey.CONTEXT_SIZE: int(credentials.get('context_size', 4096)),
ModelPropertyKey.MODE: LLMMode.CHAT.value,
},
parameter_rules=[
ParameterRule(
name='temperature',
use_template='temperature',
label=I18nObject(en_US='Temperature', zh_Hans='温度'),
type=ParameterType.FLOAT,
),
ParameterRule(
name='max_tokens',
use_template='max_tokens',
default=512,
min=1,
max=int(credentials.get('max_tokens', 4096)),
label=I18nObject(en_US='Max Tokens', zh_Hans='最大标记'),
type=ParameterType.INT,
),
ParameterRule(
name='top_p',
use_template='top_p',
label=I18nObject(en_US='Top P', zh_Hans='Top P'),
type=ParameterType.FLOAT,
),
]
)Ensure that your credentials dictionary includes 'function_calling_type': 'tool_call' when invoking the model. Here is an example of how you might set up the credentials:
credentials = {
'api_key': 'your_api_key',
'function_calling_type': 'tool_call',
'context_size': 4096,
'max_tokens': 512,
# other necessary credentials
}By setting the function_calling_type to 'tool_call', the model will recognize that it supports tool calls, and the function calls should work correctly [1].
works perfect! Thanks, dify team!
sorry，it still have some problem with Oneapi（moonshot）. if use moonshot directly ,there's no problem.
sorry，it still have some problem with Oneapi（moonshot）. if use moonshot directly ,there's no problem.
seems like lost some params when using oneapi with ToolCall
The "Invalid request: agent message content is empty" error suggests that the prompt_messages parameter might be empty or not properly populated when using OneAPI with Moonshot. Ensure that the prompt_messages list contains valid PromptMessage objects before invoking the method.
Here is an example of how to set up the prompt_messages:
from core.model_runtime.entities.message_entities import UserPromptMessage, AssistantPromptMessage
prompt_messages = [
UserPromptMessage(content="Your user prompt here"),
AssistantPromptMessage(content="Your assistant prompt here")
]
credentials = {
'api_key': 'your_api_key',
'function_calling_type': 'tool_call',
'context_size': 4096,
'max_tokens': 512,
# other necessary credentials
}
model_parameters = {
'temperature': 0.7,
'top_p': 0.9,
'max_tokens': 512
}
# Assuming `moonshot_model` is an instance of `MoonshotLargeLanguageModel`
result = moonshot_model._invoke(
model="moonshot-v1-8k",
credentials=credentials,
prompt_messages=prompt_messages,
model_parameters=model_parameters
)Ensure that the prompt_messages list is not empty and contains the necessary message content. This should help resolve the "Invalid request: agent message content is empty" error [1].
The "Invalid request: agent message content is empty" error suggests that the
prompt_messagesparameter might be empty or not properly populated when using OneAPI with Moonshot. Ensure that theprompt_messageslist contains validPromptMessageobjects before invoking the method.Here is an example of how to set up the
prompt_messages:from core.model_runtime.entities.message_entities import UserPromptMessage, AssistantPromptMessage prompt_messages = [ UserPromptMessage(content="Your user prompt here"), AssistantPromptMessage(content="Your assistant prompt here") ] credentials = { 'api_key': 'your_api_key', 'function_calling_type': 'tool_call', 'context_size': 4096, 'max_tokens': 512, # other necessary credentials } model_parameters = { 'temperature': 0.7, 'top_p': 0.9, 'max_tokens': 512 } # Assuming `moonshot_model` is an instance of `MoonshotLargeLanguageModel` result = moonshot_model._invoke( model="moonshot-v1-8k", credentials=credentials, prompt_messages=prompt_messages, model_parameters=model_parameters )Ensure that the
prompt_messageslist is not empty and contains the necessary message content. This should help resolve the "Invalid request: agent message content is empty" error [1].
I'm using dify web chat window. Not using api. the response is from dify.
by the way, Toocall response is good from Logs. 
dify should send tool response with context to oneapi(moonshot). But Oneapi response with 400(no params)
seem like interrupt by some error
I'v seen this issue is marked as [feat:model-runtime],what does that mean? should i wait for latest release?
The issue you're experiencing with the "Invalid request: agent message content is empty" error when using OneAPI with Moonshot in Dify version 0.6.11 is likely related to the prompt_messages parameter not being properly populated or formatted.
Ensure that your prompt_messages list contains valid PromptMessage objects and that they are correctly converted to the required dictionary format. Here is a snippet from the code that converts PromptMessage objects to the required dictionary format:
def _convert_prompt_message_to_dict(self, message: PromptMessage, credentials: Optional[dict] = None) -> dict:
if isinstance(message, UserPromptMessage):
message = cast(UserPromptMessage, message)
if isinstance(message.content, str):
message_dict = {"role": "user", "content": message.content}
else:
sub_messages = []
for message_content in message.content:
if message_content.type == PromptMessageContentType.TEXT:
message_content = cast(PromptMessageContent, message_content)
sub_message_dict = {
"type": "text",
"text": message_content.data
}
sub_messages.append(sub_message_dict)
elif message_content.type == PromptMessageContentType.IMAGE:
message_content = cast(ImagePromptMessageContent, message_content)
sub_message_dict = {
"type": "image_url",
"image_url": {
"url": message_content.data,
"detail": message_content.detail.value
}
}
sub_messages.append(sub_message_dict)
message_dict = {"role": "user", "content": sub_messages}
elif isinstance(message, AssistantPromptMessage):
message = cast(AssistantPromptMessage, message)
message_dict = {"role": "assistant", "content": message.content}
if message.tool_calls:
message_dict["tool_calls"] = []
for function_call in message.tool_calls:
message_dict["tool_calls"].append({
"id": function_call.id,
"type": function_call.type,
"function": {
"name": function_call.function.name,
"arguments": function_call.function.arguments
}
})
elif isinstance(message, ToolPromptMessage):
message = cast(ToolPromptMessage, message)
message_dict = {"role": "tool", "content": message.content, "tool_call_id": message.tool_call_id}
elif isinstance(message, SystemPromptMessage):
message = cast(SystemPromptMessage, message)
message_dict = {"role": "system", "content": message.content}
else:
raise ValueError(f"Got unknown type {message}")
if message.name:
message_dict["name"] = message.name
return message_dictEnsure that your prompt_messages are correctly formatted and populated before making the API call. The PromptMessage class and its subclasses (e.g., UserPromptMessage, AssistantPromptMessage, SystemPromptMessage, ToolPromptMessage) should be properly instantiated with the required attributes [1][2].
Regarding the [feat:model-runtime] label, it indicates that the issue is related to the model runtime feature. There have been recent updates in the latest version (v0.6.12-fix1) that address issues related to prompt_messages and the feat:model-runtime feature. These updates ensure that the handling of prompt_messages and the construction of parameters for the glm-4v model are more robust [3][4].
If the issue persists, it might be related to the specific implementation of the OneAPI with Moonshot, and you may need to wait for the latest release to resolve this issue, especially since it is marked as [feat:model-runtime].
The issue you're experiencing with the "Invalid request: agent message content is empty" error when using OneAPI with Moonshot in Dify version 0.6.11 is likely related to the
prompt_messagesparameter not being properly populated or formatted.Ensure that your
prompt_messageslist contains validPromptMessageobjects and that they are correctly converted to the required dictionary format. Here is a snippet from the code that convertsPromptMessageobjects to the required dictionary format:def _convert_prompt_message_to_dict(self, message: PromptMessage, credentials: Optional[dict] = None) -> dict: if isinstance(message, UserPromptMessage): message = cast(UserPromptMessage, message) if isinstance(message.content, str): message_dict = {"role": "user", "content": message.content} else: sub_messages = [] for message_content in message.content: if message_content.type == PromptMessageContentType.TEXT: message_content = cast(PromptMessageContent, message_content) sub_message_dict = { "type": "text", "text": message_content.data } sub_messages.append(sub_message_dict) elif message_content.type == PromptMessageContentType.IMAGE: message_content = cast(ImagePromptMessageContent, message_content) sub_message_dict = { "type": "image_url", "image_url": { "url": message_content.data, "detail": message_content.detail.value } } sub_messages.append(sub_message_dict) message_dict = {"role": "user", "content": sub_messages} elif isinstance(message, AssistantPromptMessage): message = cast(AssistantPromptMessage, message) message_dict = {"role": "assistant", "content": message.content} if message.tool_calls: message_dict["tool_calls"] = [] for function_call in message.tool_calls: message_dict["tool_calls"].append({ "id": function_call.id, "type": function_call.type, "function": { "name": function_call.function.name, "arguments": function_call.function.arguments } }) elif isinstance(message, ToolPromptMessage): message = cast(ToolPromptMessage, message) message_dict = {"role": "tool", "content": message.content, "tool_call_id": message.tool_call_id} elif isinstance(message, SystemPromptMessage): message = cast(SystemPromptMessage, message) message_dict = {"role": "system", "content": message.content} else: raise ValueError(f"Got unknown type {message}") if message.name: message_dict["name"] = message.name return message_dictEnsure that your
prompt_messagesare correctly formatted and populated before making the API call. ThePromptMessageclass and its subclasses (e.g.,UserPromptMessage,AssistantPromptMessage,SystemPromptMessage,ToolPromptMessage) should be properly instantiated with the required attributes [1][2].Regarding the [feat:model-runtime] label, it indicates that the issue is related to the model runtime feature. There have been recent updates in the latest version (v0.6.12-fix1) that address issues related to
prompt_messagesand thefeat:model-runtimefeature. These updates ensure that the handling ofprompt_messagesand the construction of parameters for theglm-4vmodel are more robust [3][4].If the issue persists, it might be related to the specific implementation of the OneAPI with Moonshot, and you may need to wait for the latest release to resolve this issue, especially since it is marked as [feat:model-runtime].
after upgrade to the latest version(v0.6.12-fix1)，it gave another error "lnvalid reauest: tool call id is not found"
To resolve the "Invalid request: tool_call_id is not found" error after upgrading to Dify version 0.6.12-fix1, ensure that the tool_call_id is correctly set in the ToolPromptMessage objects within your prompt_messages list. The tool_call_id must match the id of the tool call in the AssistantPromptMessage.
Here is a relevant snippet from the code that shows how tool_call_id is used:
def _convert_prompt_messages_to_message_and_chat_histories(self, prompt_messages: list[PromptMessage]) \
-> tuple[str, list[ChatMessage], list[ChatStreamRequestToolResultsItem]]:
chat_histories = []
latest_tool_call_n_outputs = []
for prompt_message in prompt_messages:
if prompt_message.role == PromptMessageRole.ASSISTANT:
prompt_message = cast(AssistantPromptMessage, prompt_message)
if prompt_message.tool_calls:
for tool_call in prompt_message.tool_calls:
latest_tool_call_n_outputs.append(ChatStreamRequestToolResultsItem(
call=ToolCall(
name=tool_call.function.name,
parameters=json.loads(tool_call.function.arguments)
),
outputs=[]
))
else:
cohere_prompt_message = self._convert_prompt_message_to_dict(prompt_message)
if cohere_prompt_message:
chat_histories.append(cohere_prompt_message)
elif prompt_message.role == PromptMessageRole.TOOL:
prompt_message = cast(ToolPromptMessage, prompt_message)
if latest_tool_call_n_outputs:
i = 0
for tool_call_n_outputs in latest_tool_call_n_outputs:
if tool_call_n_outputs.call.name == prompt_message.tool_call_id:
latest_tool_call_n_outputs[i] = ChatStreamRequestToolResultsItem(
call=ToolCall(
name=tool_call_n_outputs.call.name,
parameters=tool_call_n_outputs.call.parameters
),
outputs=[{
"result": prompt_message.content
}]
)
break
i += 1
else:
cohere_prompt_message = self._convert_prompt_message_to_dict(prompt_message)
if cohere_prompt_message:
chat_histories.append(cohere_prompt_message)
if latest_tool_call_n_outputs:
new_latest_tool_call_n_outputs = []
for tool_call_n_outputs in latest_tool_call_n_outputs:
if tool_call_n_outputs.outputs:
new_latest_tool_call_n_outputs.append(tool_call_n_outputs)
latest_tool_call_n_outputs = new_latest_tool_call_n_outputs
if len(chat_histories) > 0:
latest_message = chat_histories.pop()
message = latest_message.message
else:
raise ValueError('Prompt messages is empty')
return message, chat_histories, latest_tool_call_n_outputsAdditionally, ensure that the tool_call_id is included in the parameters when invoking the model, as shown in the _chat_generate method:
def _chat_generate(self, model: str, credentials: dict,
prompt_messages: list[PromptMessage], model_parameters: dict,
tools: Optional[list[PromptMessageTool]] = None, stop: Optional[list[str]] = None,
stream: bool = True, user: Optional[str] = None) -> Union[LLMResult, Generator]:
client = AzureOpenAI(**self._to_credential_kwargs(credentials))
response_format = model_parameters.get("response_format")
if response_format:
if response_format == "json_object":
response_format = {"type": "json_object"}
else:
response_format = {"type": "text"}
model_parameters["response_format"] = response_format
extra_model_kwargs = {}
if tools:
extra_model_kwargs['tools'] = [helper.dump_model(PromptMessageFunction(function=tool)) for tool in tools]
if stop:
extra_model_kwargs['stop'] = stop
if user:
extra_model_kwargs['user'] = user
messages = [self._convert_prompt_message_to_dict(m) for m in prompt_messages]
response = client.chat.completions.create(
messages=messages,
model=model,
stream=stream,
**model_parameters,
**extra_model_kwargs,
)
if stream:
return self._handle_chat_generate_stream_response(model, credentials, response, prompt_messages, tools)
return self._handle_chat_generate_response(model, credentials, response, prompt_messages, tools)Ensure that each ToolPromptMessage has a valid tool_call_id that corresponds to an existing tool call in the AssistantPromptMessage [1][2][3][4][5].