DLRM suggest benchmark

Question

DLRM suggest benchmark

JunxiChhen opened this issue 2 years ago · comments

Hi,
I read about your doc and the example of benchmark only has CNN related model.
And what benchmark tool do you suggest to use with DLRM and running on AMD platform using ZenDNN?

Thanks

Ratan Prasad · Answer 1 · Wed Jul 13 2022 16:37:37 GMT+0800 (China Standard Time)

ZenDNN library is intended to be used in conjunction with the frameworks like TensorFlow, PyTorch, ONNXRT and cannot be used independently.
ZenDNN release v3.3 is available at https://developer.amd.com/zendnn/

We recommend to refer the official DLRM https://github.com/facebookresearch/dlrm.git for benchmarking.

Jason Chen · Answer 2 · Wed Jul 13 2022 20:33:49 GMT+0800 (China Standard Time)

Thank you! It is really helpful. BTW, how about the suggest benchmark of Bert-Large?

Jason Chen · Answer 3 · Wed Jul 13 2022 21:51:34 GMT+0800 (China Standard Time)

ZenDNN library is intended to be used in conjunction with the frameworks like TensorFlow, PyTorch, ONNXRT and cannot be used independently. ZenDNN release v3.3 is available at https://developer.amd.com/zendnn/

We recommend to refer the official DLRM https://github.com/facebookresearch/dlrm.git for benchmarking.

When I ran the official dlrm benchmark with ZenDNN v3.3 pytorch (python3.8), there is an issue:
RuntimeError: could not execute a primitive
And when I use previous ZenDNN version (v3.3), there is a core dumped issue:
Segmentation fault (core dumped)
The issue happened when min-batch-size equal and larger than 128 on my test vm (m6a.16xlarge).

These issues didn't happen when I didn't use ZenDNN.
So are there any tricks to avoid them?

The benchmark cmd:
export OMP_NUM_THREADS=8 && export GOMP_CPU_AFFINITY=0-7 && numactl --physcpubind=0-7 -m 0 python dlrm_s_pytorch.py --mini-batch-size=128 --test-num-workers=0 --num-batches=100 --data-generation=random --arch-mlp-bot=512-512-64 --arch-mlp-top=1024-1024-1024-1 --arch-sparse-feature-size=64 --arch-embedding-size=1000000-1000000-1000000-1000000-1000000-1000000-1000000-1000000 --num-indices-per-lookup=100 --arch-interaction-op=dot --numpy-rand-seed=727 --print-freq=100 --print-time --inference-only

Jason Chen · Answer 4 · Mon Jul 25 2022 10:12:29 GMT+0800 (China Standard Time)

Segmentation fault (core dumped) occurred using ZenDNN3.2 pytorch when using criteo data benchmark on dlrm_s_pytorch.py (in https://github.com/facebookresearch/dlrm.git)

Ratan Prasad · Answer 5 · Wed Jul 27 2022 12:54:20 GMT+0800 (China Standard Time)

Thank you! It is really helpful. BTW, how about the suggest benchmark of Bert-Large?

Hi, Please refer the Bert: https://github.com/google-research/bert

Ratan Prasad · Answer 6 · Wed Jul 27 2022 12:57:25 GMT+0800 (China Standard Time)

Segmentation fault (core dumped) occurred using ZenDNN3.2 pytorch when using criteo data benchmark on dlrm_s_pytorch.py (in https://github.com/facebookresearch/dlrm.git)
Hi, we found the issue when running official dlrm benchmark and criteo data benchmark.
We will submit the fix very soon.

Jason Chen · Answer 7 · Wed Jul 27 2022 13:09:07 GMT+0800 (China Standard Time)

Segmentation fault (core dumped) occurred using ZenDNN3.2 pytorch when using criteo data benchmark on dlrm_s_pytorch.py (in https://github.com/facebookresearch/dlrm.git)
Hi, we found the issue when running official dlrm benchmark and criteo data benchmark.
We will submit the fix very soon.

Thank you. And how about the "RuntimeError: could not execute a primitive" using zendnn3.3?

Ratan Prasad · Answer 8 · Wed Jul 27 2022 13:15:33 GMT+0800 (China Standard Time)

Segmentation fault (core dumped) occurred using ZenDNN3.2 pytorch when using criteo data benchmark on dlrm_s_pytorch.py (in https://github.com/facebookresearch/dlrm.git)
Hi, we found the issue when running official dlrm benchmark and criteo data benchmark.
We will submit the fix very soon.

Thank you. And how about the "RuntimeError: could not execute a primitive" using zendnn3.3?

That is also fixed. The fix will be submitted very soon.