Local LLM option generates content repeatedly
StargazerEcho opened this issue · comments
Current main, when used with a local LLM, keeps generating the results content over and over again. Doesn't happen when using Groq,
This is largely dependent on the model you are running. Try a different model and see if you see the same results.
I am on this commit yhyu13@378c5de
Can confirm this issue with this model dolphin 2.9.1 Yi-1.5 34B 8bit (https://huggingface.co/crusoeai/dolphin-2.9.1-yi-1.5-34b-GGUF/tree/main) can responde with tool calling, with llamacpp openai local server. But generate following output in the backend
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
openai.base_url is http://127.0.0.1:5051/v1
INFO: Started server process [267929]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: 127.0.0.1:35020 - "OPTIONS /chat HTTP/1.1" 200 OK
INFO: 127.0.0.1:35020 - "POST /chat HTTP/1.1" 200 OK
Error code: 500 - {'error': {'code': 500, 'message': 'Unsupported param: tools', 'type': 'server_error'}}
INFO: 127.0.0.1:35020 - "POST /chat HTTP/1.1" 200 OK
Error code: 500 - {'error': {'code': 500, 'message': 'Unsupported param: tools', 'type': 'server_error'}}
INFO: 127.0.0.1:35020 - "POST /chat HTTP/1.1" 200 OK
Error code: 500 - {'error': {'code': 500, 'message': 'Unsupported param: tools', 'type': 'server_error'}}
INFO: 127.0.0.1:35020 - "POST /chat HTTP/1.1" 200 OK
Error code: 500 - {'error': {'code': 500, 'message': 'Unsupported param: tools', 'type': 'server_error'}}
INFO: 127.0.0.1:35020 - "POST /chat HTTP/1.1" 200 OK
Error code: 500 - {'error': {'code': 500, 'message': 'Unsupported param: tools', 'type': 'server_error'}}
INFO: 127.0.0.1:35020 - "POST /chat HTTP/1.1" 200 OK
Error code: 500 - {'error': {'code': 500, 'message': 'Unsupported param: tools', 'type': 'server_error'}}
INFO: 127.0.0.1:35020 - "POST /chat HTTP/1.1" 200 OK
Error code: 500 - {'error': {'code': 500, 'message': 'Unsupported param: tools', 'type': 'server_error'}}
INFO: 127.0.0.1:35020 - "POST /chat HTTP/1.1" 200 OK
Error code: 500 - {'error': {'code': 500, 'message': 'Unsupported param: tools', 'type': 'server_error'}}
INFO: 127.0.0.1:35020 - "POST /chat HTTP/1.1" 200 OK
Error code: 500 - {'error': {'code': 500, 'message': 'Unsupported param: tools', 'type': 'server_error'}}
INFO: 127.0.0.1:35020 - "POST /chat HTTP/1.1" 200 OK
^C^CINFO: Shutting down
Here is the repeated output by the model in the client side (which repeat on citation, and I have to stop it...)
Farfalle is a type of Italian pasta known for its distinctive butterfly or bow tie shape. It is made from a dough of semolina flour, all-purpose flour, and eggs, which is rolled out thin and cut into small rectangles. The pasta's signature ruffled edges are achieved using pinking shears. The name "farfalle" comes from the Italian word for "butterflies." In the Italian region of Emilia-Romagna, farfalle are also known as strichetti, a local term for "bow ties." Larger variations of farfalle are called farfalloni, while the miniature version is called farfalline. Farfalle can also be made from wholemeal flour and may have natural dyes added to the dough to change its color and taste. This pasta dates back to the 16th century in Lombardia and Emilia-Romagna Regions in Northern Italy and is often used with cream sauces and in tomato-based sauces.
Citations:
https://www.delallo.com/blog/what-is-farfalle
https://en.wikipedia.org/wiki/Farfalle
https://www.thespruceeats.com/what-is-farfalle-995663
https://pastamacaroni.com/farfalle/
https://pastaevangelists.com/blogs/blog/how-to-make-farfalle-pasta-at-home
https://juliasalbum.com/farfalle-with-spinach-mushrooms-caramelized-onions/<|im_end|>Farfalle is an Italian, bow tie shaped small pasta with either straight or frilly edges. It is made from a dough of semolina flour, all-purpose flour, and eggs. The dough is rolled out thin and then cut into small rectangles that are about one and one-half by one inch. To get the pasta's signature ruffled edges, a pair of pinking shears is used
. Farfalle, also known as "butterflies," are a type of pasta with a name derived from the Italian word farfalle. In the Italian region of Emilia-Romagna, farfalle are known as strichetti, a local word for 'bow ties'. A larger variation of farfalle is known as farfalloni, while the miniature version is called farfalline
. Farfalle integrale is a type of farfalle pasta that is made from wholemeal rather than refined white flour, with natural dyes such as beets, spinach, cuttlefish ink, or different kinds of flour or seasonings added to the dough to change the color and taste of the pasta
. Farfalle is a popular pasta cut that is named for its "butterfly" shape. Like most Italian pasta, Farfalle is made with durum wheat flour, or semolina, and water that is kneaded together to create the pasta dough. The pasta dough is then rolled into sheets. To form its distinct shape, the pasta sheet is cut into small rectangles with a zig
. Farfalle are often described as having a bow-tie or butterfly shape. In fact, the word 'farfalle' is the Italian word for "butterflies." Farfalle (pronounced far-fall-ay) are one of Italy's oldest pasta shapes, dating back as far as the [1](https://pastamacaroni.com/farfalle/)6th century
. Farfalle is a bow-tie shaped Italian pasta. Farfalle pasta can also be described as butterfly-shaped pasta. In fact, "farfalle" means "butterflies" in Italian. Farfalle dates back to the 16th century in Lombardia and Emilia-Romagna Regions in Northern Italy. Farfalle is often used with cream sauces and in tomato-based sauces
.<|im_end|>Farfalle is a type of Italian pasta known for its distinctive bow-tie or butterfly shape. The name "farfalle" comes from the Italian word for "butterflies." This pasta dates back to the 16th century and is made from a dough of semolina flour, all-purpose flour, and eggs. The dough is rolled out, cut into small rectangles, and then given its signature ruffled edges using pinking shears. Farfalle is versatile and can be used in various dishes, such as cream sauces and tomato-based sauces.
Citations:
,
,
,
<|im_end|>Farfalle is a type of Italian pasta known for its distinctive bow-tie or butterfly shape. The name "farfalle" comes from the Italian word for "butterflies." This pasta dates back to the 16th century and is made from a dough of semolina flour, all-purpose flour, and eggs. The dough is rolled out thin, cut into small rectangles, and then given its signature ruffled edges using pinking shears. Farfalle is versatile and can be used in various dishes, including cream sauces and tomato-based sauces.
Citations:
https://pastaevangelists.com/blogs/blog/how-to-make-farfalle-pasta-at-home
https://www.thespruceeats.com/what-is-farfalle-99566[3](https://www.thespruceeats.com/what-is-farfalle-995663)
https://en.wikipedia.org/wiki/Farfalle
https://www.delallo.com/blog/what-is-farfalle
https://juliasalbum.com/farfalle-with-spinach-mushrooms-caramelized-onions/<|im_end|>Farfalle is a type of Italian pasta known for its distinctive butterfly or bow tie shape. It is made from a dough of durum wheat flour, or semolina, and water. The pasta dough is rolled into sheets, which are then cut into small rectangles with a zig-zag pattern to create its unique shape. Farfalle is also referred to as "butterflies" in Italian, and it can be found in different sizes, such as farfalloni (larger) and farfalline (miniature). The name "farfalle" is derived from the Italian word for "butterflies." In the Italian region of Emilia-Romagna, farfalle is also known as strichetti, meaning "bow ties." Farfalle can be made with wholemeal flour, and natural dyes such as beets, spinach, cuttlefish ink, or different kinds of flour or seasonings can be added to the dough to change the color and taste of the pasta. Farfalle dates back to the 16th century in Lombardia and Emilia-Romagna Regions in Northern Italy. It is often used with cream sauces and in tomato-based sauces.<|im_end|>Farfalle is a type of pasta that is known for its distinctive butterfly or bow-tie shape. It is made from a dough of semolina flour, all-purpose flour, and eggs, which is rolled out thin and cut into small rectangles. To create the signature ruffled edges, pinking shears are used to cut the dough. The name "farfalle" comes from the Italian word for "butterflies." In some regions of Italy, farfalle is also known as "strichetti" or "bow ties." There are variations in size, with larger versions called "farfalloni" and miniature versions called "farfalline." Farfalle is a versatile pasta that can be used in a variety of dishes, including cream sauces and tomato-based sauces.
Citations:
,
,
<|im_end|>Farfalle is a type of Italian pasta known for its butterfly or bow tie shape. It is made from a dough of semolina flour, all-purpose flour, and eggs, which is rolled out thin, cut into small rectangles, and then given its signature ruffled edges using pinking shears. The name farfalle comes from the Italian word for butterflies, and it is also referred to as strichetti, meaning 'bow ties' in the Emilia-Romagna region of Italy. Farfalle is versatile and can be used in pasta salads, with heavy red or cream-based sauces, or simply tossed with olive oil or butter. There are variations in size, such as farfalloni for larger pasta and farfalline for a smaller version. Additionally, farfalle integrale is made from wholemeal flour and can be colored and flavored using natural dyes like beets, spinach, or cuttlefish ink.<|im_end|>Farfalle is a type of Italian pasta known for its distinctive bow-tie or butterfly shape. It is made from a dough of semolina flour, all-purpose flour, and eggs, which is rolled out thin, cut into small rectangles, and then given its signature ruffled edges using pinking shears. The name "farfalle" comes from the Italian word for "butterflies," and the pasta dates back to the 16th century. There are variations such as farfalloni (larger) and farfalline (miniature), as well as farfalle integrale, which is made from wholemeal flour and can be colored with natural dyes like beets, spinach, or cuttlefish ink. Farfalle is versatile and can be used with cream sauces or in tomato-based sauces.
Citations:
Title: What Is Farfalle? - The Spruce Eats
Title: Farfalle Recipe: Everything You Need to Know | Pasta Evangelists
Title: Spotlight Series: All About Farfalle (Also Known As 'Bow Tie Pasta')
Title: Farfalle Pasta | Best bow tie (Butterfly) Pasta Recipes and more ...
Title: Farfalle - Wikipedia
Title: Farfalle with Spinach, Mushrooms, Caramelized Onions<|im_end|>Farfalle is an Italian pasta with a distinctive bow tie or butterfly shape. It is made from a dough of semolina flour, all-purpose flour, and eggs, which is rolled out thin, cut into small rectangles, and then given its signature ruffled edges using pinking shears. The name "farfalle" comes from the Italian word for "butterflies." In the Italian region of Emilia-Romagna, farfalle are also known as strichetti, meaning "bow ties." There are variations in size, with larger versions called farfalloni and miniature versions called farfalline. Farfalle dates back to the 16th century in Lombardia and Emilia-Romagna Regions in Northern Italy and is often used with cream sauces and in tomato-based sauces.
Citations:
https://www.thespruceeats.com/what-is-farfalle-99[5](https://www.delallo.com/blog/what-is-farfalle)663
https://en.wikipedia.org/wiki/Farfalle
https://pastaevangelists.com/blogs/blog/how-to-make-farfalle-pasta-at-home
https://www.delallo.com/blog/what-is-farfalle
https://juliasalbum.com/farfalle-with-spinach-mushrooms-caramelized-onions/<|im_end|>
Solved guys! It has to do with llm inference server and your model.
I have succeeded with this backend server and its coresponding models fine tuned for function calling https://github.com/MeetKai/functionary
Here is local script
#!/bin/bash
eval "$(conda shell.bash hook)"
conda activate functionary
#NAME=functionary-medium-v2.4-AWQ
NAME=functionary-small-v2.5
TL_PL=1
model_path=meetkai/$NAME
model_names="gpt-4o"
CUDA_VISIBLE_DEVICES=0 python3 server_vllm.py --model $model_path --host 127.0.0.1 --port 5051 --max-model-len 8192 --served-model-name $model_names --tensor-parallel-size $TL_PL