Numerical issue with qwen7b.vmfb model

[Manish Kumar]

What happened?

I'm able to compile the pytorch model into mlir & then convert the mlir model into vmfb file:
I used this code for compilation : https://gist.github.com/manishghop/d4e51bb95c491229c5d94b4c7ca04491

But while running the inference I get NaN error:

I used this code to run the inference : https://gist.github.com/manishghop/fb3ae898b4ea2c8c6bb404efac408c0e

Steps to reproduce your issue

1. git clone https://github.com/nod-ai/SHARK.git
2. cd SHARK
3. Run the following in Powershell
   3.1. set-executionpolicy remotesigned
   3.2. Run the setup_venv.ps1 from: https://github.com/nod-ai/SHARK
4. Run qwen_compile.py to generate .vmfb file.
5. Run qwen_runner.py to run the inference.

What component(s) does this issue relate to?

Runtime

Version information

No response

Additional context

No response

[louwangzhiyuY]

Hi
did you solve the issue?

[Chi_Liu]

related issue chatglm #15661 solution in chatglm issue's comment

[Chi_Liu]

Bug log: Qwen_fail_log_20131214.txt

[Chi_Liu]

The issue should be fixed with TOM shark after the fix nod-ai/SHARK#2047

(shark.venv) ➜  nan git:(main) ✗ python qwen_runner.py
/home/chi/src/SHARK/shark.venv/lib/python3.11/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
tensor([[108386]]) torch.Size([1, 1])
/home/chi/src/SHARK/nan/qwen_runner.py:13: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  input_ids = torch.tensor(input_ids).reshape([1, input_id_len])
Loading module qwen.vmfb...
Successfully Loaded vmfb model
[[[ 0.0988 -0.4678 -1.136  ...  0.9785  0.9785  0.9785]]]