Giters

NVIDIA / FasterTransformer

Transformer related optimization, including BERT, GPT

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

TP=2， Loss of accuracy

coderchem opened this issue · comments

commented

hello,I carried out TP=2,multi-gpu operation on llama model of 7b, and found that the comparison accuracy of the result was lost by 5%. TP=2 as far as I know should not change the accuracy. Why?

commented

hi, can you post reproduce step.

commented

FasterTransformer does not support llama officially and FasterTransformer development has transitioned to TensorRT-LLM. TensorRT-LLM has supported LLaMa, please take a try.