Giters

armbues / SiLLM

SiLLM simplifies the process of training and running Large Language Models (LLMs) on Apple Silicon by leveraging the MLX framework.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Phi 3 4k , 128k mot working

kishoretvk opened this issue · comments

commented

In chat model they error out.

commented

Just confirmed that Phi-3-mini-4k-instruct is working with the latest release version 0.1.3 of SiLLM from pip.
There is an error IndexError: piece id is out of range. when trying to load the sentencepiece tokenizer for Phi-3 at this point. You can just delete the file tokenizer.model if you have downloaded tokenizer.json and tokenizer_config.json in the model directory.

The 128k version is using a custom scaled rotary embedding that I didn't port to SiLLM yet. Hopefully will have some time to take a look in the next days.
Here's the original source that needs to be ported:
SuScaledRotaryEmbedding
YarnScaledRotaryEmbedding

commented

thanks a lot and for 128k as well same error shows up at 4k one,
one more thing, DPO mistral 7b instruct v2 also give error.

commented

Just confirmed that Phi-3-mini-4k-instruct is working with the latest release version 0.1.3 of SiLLM from pip. There is an error IndexError: piece id is out of range. when trying to load the sentencepiece tokenizer for Phi-3 at this point. You can just delete the file tokenizer.model if you have downloaded tokenizer.json and tokenizer_config.json in the model directory.

The 128k version is using a custom scaled rotary embedding that I didn't port to SiLLM yet. Hopefully will have some time to take a look in the next days. Here's the original source that needs to be ported: SuScaledRotaryEmbedding YarnScaledRotaryEmbedding

great.

commented

Could you share the error message and a link to the model that doesn't work?