Can't get the FIM to work with LM Studio
Wyzix33 opened this issue · comments
Describe the bug
I've tried to use WizardCoder for FIM and it's not working, here is the log for LM Studio:
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] Success! HTTP server listening on port 11434
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] Supported endpoints:
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] -> GET http://localhost:11434/v1/models
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] -> POST http://localhost:11434/v1/chat/completions
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] -> POST http://localhost:11434/v1/completions
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] -> POST http://localhost:11434/v1/embeddings <------------ NEW!
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] Model loaded: mike-ravkine/WizardCoder-15B-V1.0-GGUF/WizardCoder-15B-V1.0.Q5_1.gguf
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] Logs are saved into C:\tmp\lmstudio-server-log.txt
[2024-04-21 11:41:23.856] [INFO] [LM STUDIO SERVER] Processing queued request...
[2024-04-21 11:41:23.857] [INFO] Received POST request to /v1/completions with body: {
"model": "WizardCoder",
"prompt": "<PRE> /**/\n\n/* Language: PHP (php) */\n/* File uri: file:///c%3A/github/magura/php/nomenclator/parteneri/clienti.php (php) */\n $sql = \"SELECT * FROM clienti\";\n$result = $conn->query($sql);\n$clienti = [];\nwhile ($row = $result->fetch_assoc()) {\n $clienti[] = $row;\n}\n$sql = \"SELECT * FROM clienti_sedii\";\n$result = $conn->query($sql);\n$sedii = [];\nwhile ($row = $result->fetch_assoc()) {\n $sedii[] = $row;\n}\nheader(\"Content-Type: application/json\");\necho json_encode(['clienti' => $clienti, 'sedii' => <SUF> \n <MID>",
"stream": true,
"temperature": 0.2,
"n_predict": 512
}
[2024-04-21 11:41:23.858] [INFO] Provided inference configuration: {
"temp": 0.2
}
[2024-04-21 11:41:23.859] [INFO] Streaming response..
[2024-04-21 11:41:23.860] [INFO] [LM STUDIO SERVER] Processing...
[2024-04-21 11:41:24.520] [INFO] [LM STUDIO SERVER] Finished streaming response
To Reproduce
start a server with any coder LLM and test, sometimes i get a completion but it's nonsense...
Expected behavior
A completion based on the context... in my case it should fill "$sedii]);" but sometimes i get nothing, other times i get some "#import <...>"
Desktop (please complete the following information):
- OS: Windows
- Version 11
Additional context
I used twinny v3.11.9
and VSCode Version: 1.88.1 (user setup)
Commit: e170252f762678dec6ca2cc69aba1570769a5d39
Date: 2024-04-10T17:41:02.734Z
Electron: 28.2.8
ElectronBuildId: 27744544
Chromium: 120.0.6099.291
Node.js: 18.18.2
V8: 12.0.267.19-electron.0
OS: Windows_NT x64 10.0.22631
Chat seems to work tho...
Thanks
Hello, it seems you are using WizardCoder for FIM but with codellama template. I don't know which fim template WizardCoder supports or if it supports FIM. You should try a different model. Either deepseek base or codellama-code.
https://github.com/rjmacarthy/twinny/blob/main/docs/providers.md#fim-auto-complete-2
I see here: https://huggingface.co/epinnock/wizardcoder-1b-merged#fill-in-the-middle that WizardCoder uses the same FIM template as Starcoder, please try that as the FIM template, or use custom and edit your own.
See docs for recommended settings.
thanks for the answer, I've made progress but still not good enough, at least now i can get the FIM to work, here is what i get, I will try other LLM maybe the ones i got are not good...
Sequence.01_1.mp4
Hey, starcoder2 has an issue with completions, I have not been successful with it either. Please try codellama or deepseek-coder. Many thanks.