RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

[LI Wentong]

Hello@jshilong, have you encountered this problem?

I have trained the model of both two stages. Then I merge the trained model with llama as you described.
When I load the merged model to do test, the errors in below occured.

Traceback (most recent call last):
  File "/hy/code/gpt4roi/train_net.py", line 326, in <module>
    launch(
  File "/hy/code/gpt4roi/detectron2/detectron2/engine/launch.py", line 84, in launch
    main_func(*args)
  File "/hy/code/gpt4roi/train_net.py", line 311, in main
    res = Trainer.test(cfg, model)
  File "/hy/code/gpt4roi/detectron2/detectron2/engine/defaults.py", line 617, in test
    results_i = inference_on_dataset(model, data_loader, evaluator)
  File "/hy/code/gpt4roi/detectron2/detectron2/evaluation/evaluator.py", line 158, in inference_on_dataset
    outputs = model(inputs)
  File "/workspace/conda_env/gpt4roi/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/hy/code/gpt4roi/gpt4roi.py", line 153, in get_output
    output_ids = self.model.generate(
  File "/workspace/conda_env/gpt4roi/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/workspace/conda_env/gpt4roi/lib/python3.10/site-packages/transformers/generation/utils.py", line 1485, in generate
    return self.sample(
  File "/workspace/conda_env/gpt4roi/lib/python3.10/site-packages/transformers/generation/utils.py", line 2562, in sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

[LI Wentong]

I have tested to merge your provided debug weights, and then load model to do inference, the same error occured.

[Shilong Zhang]

Hi, it seems there is a misunderstanding regarding the purpose of merging the weights. The reason for merging the weights is to obtain the actual LlaVA (due to the license of Llama1) in order to train the GPT4ROI model for both stages. It appears that you are loading and training the delta, and then merging the Llama, which is an incorrect process.

By the way, the debug weight I provided earlier is a merged weight of GPT4ROI for stage 2, which you can load and fine-tune directly

[LI Wentong]

@jshilong Got it. Thanks for your reply.