Code: 1001, Text = openai embedding request error

Question

Code: 1001, Text = openai embedding request error

elvin-du opened this issue 10 months ago · comments

when I run ./target/release/askbend -c askbend.toml --rebuild cmd , I got the error as blow. how to fix this?

[2023-10-16T02:07:41Z INFO ] embedding 605/1936,  time: 513.095121ms
Error: APIError: ResponseError with 1001: openai embedding request error:Internal. Code: 1001, Text = openai embedding request error: ApiError("{\"error\":{\"message\":\"This model's maximum context length is 8191 tokens, however you requested 8916 tokens (8916 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.\",\"type\":\"invalid_request_error\",\"param\":null,\"code\":null}}").

Bohu · Answer 1 · Mon Oct 16 2023 11:10:24 GMT+0800 (China Standard Time)

context length is 8191 tokens please split your content to more small piece.

Elvin · Answer 2 · Mon Oct 16 2023 11:43:44 GMT+0800 (China Standard Time)

content means the content of one file? I just put 390 files on the path which set in config file.

Bohu · Answer 3 · Mon Oct 16 2023 16:29:42 GMT+0800 (China Standard Time)

One of your file length is larger than 8191(prompts length + content length = 8916).
We should split this file into two parts, make sure not near 8191.