Benchmarking Phi-3 on CPU
bkaruman opened this issue · comments
Hi,
Is there a tool to benchmark Phi-3 model performance on CPU?
Yes, we have a benchmarking tool here: https://github.com/microsoft/onnxruntime-genai/blob/main/benchmark/python/benchmark_e2e.py
@natke
This seems to be failing for Phi-3 model from HF - https://huggingface.co/microsoft/Phi-3-mini-128k-instruct-onnx/tree/main/cpu_and_mobile/cpu-int4-rtn-block-32
I think the script needs to be updated.
Could you change the line 21 in benchmark_phi3.py to
params.set_search_options(**{...})
@aciddelgado Can you please take a look?
I think the script needs to be updated.
Could you change the line 21 in benchmark_phi3.py to
params.set_search_options(**{...})
That doesn't work either
Are you using genai installed from pip or building from main? It looks like you are using an outdated version of the benchmarking script as I was able to run the phi-3 mini model from hf with the latest version of the benchmarking script found in the main branch of this repo. Also, did you make any further changes to benchmark_e2e.py to create your benchmark_phi3.py script?
Are you using genai installed from pip or building from main? It looks like you are using an outdated version of the benchmarking script as I was able to run the phi-3 mini model from hf with the latest version of the benchmarking script found in the main branch of this repo. Also, did you make any further changes to benchmark_e2e.py to create your benchmark_phi3.py script?
Moving to the latest commit worked. benchmark_phi3.py is same as the benchmarking script from the repo.
Great! Let us know how you get on.