rmihaylov / falcontune

Tune any FALCON in 4-bit

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Inference not accurate

Hamad-exarta opened this issue · comments

I used the same dataset provided to fine tune , but the inference seems to give general replies and not from the dataset used for fine tuning.

Any Ideas?

"instruction": "What are the three primary colors?",
"input": "",
"output": "The three primary colors are red, blue, and yellow. These colors are called primary because they cannot be created by mixing other colors and all other colors can be made by combining them in various proportions. In the additive color system, used for light, the primary colors are red, green, and blue (RGB)."

The above was part of the dataset , but when i run inference i get the following

_**Enter new instruction: Describe the structure of an atom.

An atom consists of a nucleus, made up of positively charged protons and neutral electrons, and a number of electrons arranged in shells or layers around the nucleus. The number of electrons in the outermost shell of the nucleus determines the atom's electronic**_

i used the following generate script

falcontune generate
--interactive
--model falcon-7b-instruct-4bit
--weights ./gptq_model-4bit-64g.safetensors
--lora_apply_dir falcon-7b-instruct-4bit-alpaca-selftrained/
--max_new_tokens 50
--use_cache
--do_sample
--instruction "What are the three primary colors?"
--backend triton