RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

[Chengyuan Li]

Great work!

I tried your example for llama-7b-chat and changed the tree structure in choices.py into baseline = [[0], [0, 0], [0, 0, 0], [0, 0, 0, 0]] to simulate speculative decoding without token tree, and got the error above. Any clues?

log attached:

[yuhuili]

This seems to be an issue with the original LLaMA2-Chat encountering NaN. What was your input?

[Chengyuan Li]

This seems to be an issue with the original LLaMA2-Chat encountering NaN. What was your input?

Simply "Hello". @Liyuhui-12

[yuhuili]

I can run this code normally. Did you use the correct weights for LLaMA2-Chat? It should be meta-llama/Llama-2-7b-chat-hf instead of meta-llama/Llama-2-7b-chat.

[Chengyuan Li]

I got the tree shape wrong, my bad. Thanks!

[Reyna Abhyankar]

What was wrong with your original tree shape @cyLi-Tiger? I am facing a similar issue