[TorchBench] Performance Signal Detected

Question

[TorchBench] Performance Signal Detected

zzpmiracle opened this issue a year ago · comments

zzp_miracle commented a year ago

TorchBench CI has detected a performance signal.

Affected Tests:

eval-cuda-fp32:
- attention_is_all_you_need_pytorch[dynamo-blade (latency)] 6.6 -> 5.919, +10.3182%
- attention_is_all_you_need_pytorch[dynamo-disc (latency)] 5.587 -> 4.908, +12.1532%
- DALLE2_pytorch[disc (latency)] status changed, 90.621 -> OSError
- DALLE2_pytorch[disc (compiled)] status changed, 22800.0 -> N/A
- DALLE2_pytorch[disc (clusters)] status changed, 184.0 -> N/A
- detectron2_fasterrcnn_r_101_c4[dynamo-blade (latency)] status changed, AssertionError -> 81.218
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (latency)] status changed, AssertionError -> 144.273
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (clusters)] status changed, N/A -> 15.0
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (compiled)] status changed, N/A -> 1445.0
- detectron2_fasterrcnn_r_101_dc5[dynamo-blade (latency)] status changed, AssertionError -> 42.698
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (latency)] status changed, AssertionError -> 53.583
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (clusters)] status changed, N/A -> 14.0
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (compiled)] status changed, N/A -> 1453.0
- detectron2_fasterrcnn_r_101_fpn[dynamo-blade (latency)] status changed, AssertionError -> 31.027
- detectron2_fasterrcnn_r_50_c4[dynamo-blade (latency)] status changed, AssertionError -> 76.368
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (latency)] status changed, AssertionError -> 33.85
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (clusters)] status changed, N/A -> 16.0
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (compiled)] status changed, N/A -> 958.0
- detectron2_fasterrcnn_r_50_dc5[dynamo-blade (latency)] status changed, AssertionError -> 38.006
- detectron2_fasterrcnn_r_50_fpn[dynamo-blade (latency)] status changed, AssertionError -> 25.141
- detectron2_maskrcnn_r_101_c4[dynamo-blade (latency)] status changed, AssertionError -> 90.089
- detectron2_maskrcnn_r_50_c4[dynamo-blade (latency)] status changed, AssertionError -> 84.22
- detectron2_maskrcnn_r_50_fpn[dynamo-blade (latency)] status changed, AssertionError -> 29.704
- dlrm[disc (latency)] 1.861 -> 2.125, -14.1859%
- dlrm[blade (latency)] 1.681 -> 1.926, -14.5747%
- dlrm[dynamo-blade (latency)] 1.784 -> 2.12, -18.8341%
- dlrm[dynamo-disc (latency)] 1.9 -> 2.135, -12.3684%
- drq[dynamo-blade (latency)] status changed, 1.4 -> UnserializableException
- drq[dynamo-disc (latency)] status changed, 1.353 -> UnserializableException
- drq[dynamo-disc (clusters)] status changed, 1.0 -> N/A
- drq[dynamo-disc (compiled)] status changed, 84.0 -> N/A
- fambench_xlmr[dynamo-blade (latency)] 247.478 -> 145.269, +41.3002%
- fambench_xlmr[dynamo-disc (latency)] status changed, 174.359 -> OSError
- fambench_xlmr[dynamo-disc (clusters)] status changed, 50.0 -> N/A
- fambench_xlmr[dynamo-disc (compiled)] status changed, 2868.0 -> N/A
- functorch_maml_omniglot[dynamo-disc (latency)] 0.663 -> 0.534, +19.457%
- hf_Bart[dynamo-blade (latency)] 12.191 -> 9.748, +20.0394%
- hf_Bart[dynamo-disc (latency)] 12.929 -> 10.224, +20.922%
- hf_Bart[dynamo-disc (clusters)] 8 -> 1
- hf_Bart[dynamo-disc (compiled)] 1414 -> 1426
- hf_Bert[dynamo-blade (latency)] 8.03 -> 6.977, +13.1133%
- hf_Bert[dynamo-disc (latency)] 8.199 -> 7.293, +11.0501%
- hf_Bert_large[dynamo-blade (latency)] 19.724 -> 18.105, +8.2083%
- hf_Bert_large[dynamo-disc (latency)] 21.283 -> 19.619, +7.8184%
- hf_Bert_mini[blade (latency)] 0.653 -> 0.547, +16.2328%
- hf_Bert_mini[dynamo-blade (latency)] 1.025 -> 0.575, +43.9024%
- hf_Bert_mini[dynamo-disc (latency)] 1.566 -> 0.884, +43.5504%
- hf_BigBird[disc (latency)] status changed, 160.796 -> RuntimeError
- hf_BigBird[dynamo-blade (latency)] status changed, OSError -> 116.547
- hf_BigBird[dynamo-disc (latency)] status changed, RuntimeError -> 113.666
- hf_BigBird[disc (compiled)] status changed, 5007.0 -> N/A
- hf_BigBird[disc (clusters)] status changed, 61.0 -> N/A
- hf_BigBird[dynamo-disc (clusters)] status changed, N/A -> 135.0
- hf_BigBird[dynamo-disc (compiled)] status changed, N/A -> 11446.0
- hf_DistilBert[dynamo-blade (latency)] 4.095 -> 3.755, +8.3028%
- hf_DistilBert[dynamo-disc (latency)] 4.298 -> 3.911, +9.0042%
- hf_Longformer[disc (latency)] status changed, 137.037 -> RuntimeError
- hf_Longformer[dynamo-disc (latency)] status changed, 119.214 -> RuntimeError
- hf_Longformer[disc (compiled)] status changed, 9201.0 -> N/A
- hf_Longformer[disc (clusters)] status changed, 169.0 -> N/A
- hf_Longformer[dynamo-disc (clusters)] status changed, 172.0 -> N/A
- hf_Longformer[dynamo-disc (compiled)] status changed, 6130.0 -> N/A
- phlippe_densenet[dynamo-blade (latency)] 6.578 -> 6.187, +5.9441%
- pyhpc_isoneutral_mixing[blade (latency)] 11.621 -> 10.666, +8.2179%
- pyhpc_isoneutral_mixing[dynamo-blade (latency)] 10.665 -> 9.687, +9.1702%
- pyhpc_turbulent_kinetic_energy[dynamo-blade (latency)] 10.589 -> 11.729, -10.7659%
- resnet18[dynamo-blade (latency)] 2.185 -> 2.044, +6.4531%
- timm_efficientdet[blade (latency)] status changed, 1055.4 -> RuntimeError
- timm_vision_transformer[dynamo-disc (latency)] 7.076 -> 6.682, +5.5681%
- timm_vovnet[dynamo-blade (latency)] 22.05 -> 20.386, +7.5465%
- yolov3[dynamo-disc (latency)] status changed, 44.194 -> RuntimeError
- yolov3[dynamo-disc (clusters)] status changed, 18.0 -> N/A
- yolov3[dynamo-disc (compiled)] status changed, 141.0 -> N/A
eval-cuda-fp16:
- attention_is_all_you_need_pytorch[dynamo-blade (latency)] 5.219 -> 4.503, +13.7191%
- attention_is_all_you_need_pytorch[dynamo-disc (latency)] 3.67 -> 2.664, +27.4114%
- detectron2_fasterrcnn_r_101_c4[dynamo-blade (latency)] status changed, AssertionError -> 49.041
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (latency)] status changed, AssertionError -> 58.52
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (clusters)] status changed, N/A -> 15.0
- detectron2_fasterrcnn_r_101_c4[dynamo-disc (compiled)] status changed, N/A -> 1445.0
- detectron2_fasterrcnn_r_101_dc5[dynamo-blade (latency)] status changed, AssertionError -> 27.249
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (latency)] status changed, AssertionError -> 30.856
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (clusters)] status changed, N/A -> 14.0
- detectron2_fasterrcnn_r_101_dc5[dynamo-disc (compiled)] status changed, N/A -> 1453.0
- detectron2_fasterrcnn_r_101_fpn[dynamo-blade (latency)] status changed, AssertionError -> 19.129
- detectron2_fasterrcnn_r_50_c4[dynamo-blade (latency)] status changed, AssertionError -> 47.246
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (latency)] status changed, AssertionError -> 18.075
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (clusters)] status changed, N/A -> 16.0
- detectron2_fasterrcnn_r_50_c4[dynamo-disc (compiled)] status changed, N/A -> 958.0
- detectron2_fasterrcnn_r_50_dc5[dynamo-blade (latency)] status changed, AssertionError -> 24.531
- detectron2_fasterrcnn_r_50_fpn[dynamo-blade (latency)] status changed, AssertionError -> 16.575
- detectron2_maskrcnn_r_101_c4[dynamo-blade (latency)] status changed, AssertionError -> 53.017
- detectron2_maskrcnn_r_50_c4[dynamo-blade (latency)] status changed, AssertionError -> 49.022
- detectron2_maskrcnn_r_50_fpn[dynamo-blade (latency)] status changed, AssertionError -> 18.939
- dlrm[disc (latency)] 1.132 -> 1.592, -40.636%
- dlrm[blade (latency)] 1.144 -> 1.627, -42.2203%
- dlrm[dynamo-blade (latency)] 1.22 -> 1.669, -36.8033%
- dlrm[dynamo-disc (latency)] 1.225 -> 1.655, -35.102%
- fambench_xlmr[dynamo-disc (latency)] status changed, 90.117 -> OSError
- fambench_xlmr[dynamo-disc (clusters)] status changed, 50.0 -> N/A
- fambench_xlmr[dynamo-disc (compiled)] status changed, 2868.0 -> N/A
- functorch_dp_cifar10[disc (latency)] 1.704 -> 1.818, -6.6901%
- hf_Bart[dynamo-disc (latency)] 7.154 -> 6.667, +6.8074%
- hf_Bart[dynamo-disc (clusters)] 13 -> 7
- hf_Bart[dynamo-disc (compiled)] 1397 -> 1403
- hf_Bert[dynamo-blade (latency)] 4.5 -> 3.279, +27.1333%
- hf_Bert[dynamo-disc (latency)] 4.563 -> 3.297, +27.7449%
- hf_Bert_large[dynamo-blade (latency)] 10.575 -> 8.546, +19.1868%
- hf_Bert_large[dynamo-disc (latency)] 10.625 -> 9.062, +14.7106%
- hf_Bert_mini[dynamo-blade (latency)] 0.961 -> 0.504, +47.5546%
- hf_Bert_mini[dynamo-disc (latency)] 1.374 -> 0.787, +42.722%
- hf_BigBird[dynamo-disc (latency)] status changed, RuntimeError -> 67.807
- hf_BigBird[dynamo-disc (clusters)] status changed, N/A -> 135.0
- hf_BigBird[dynamo-disc (compiled)] status changed, N/A -> 11446.0
- hf_DistilBert[disc (latency)] 2.592 -> 2.736, -5.5556%
- hf_DistilBert[dynamo-blade (latency)] 3.234 -> 3.0, +7.2356%
- hf_DistilBert[dynamo-disc (latency)] 2.936 -> 2.573, +12.3638%
- hf_GPT2[dynamo-blade (latency)] 14.768 -> 13.706, +7.1912%
- hf_GPT2[dynamo-disc (latency)] 11.466 -> 10.78, +5.9829%
- hf_Longformer[disc (latency)] status changed, 85.172 -> RuntimeError
- hf_Longformer[dynamo-blade (latency)] 88.71 -> 107.848, -21.5737%
- hf_Longformer[dynamo-disc (latency)] status changed, 67.529 -> RuntimeError
- hf_Longformer[disc (compiled)] status changed, 9201.0 -> N/A
- hf_Longformer[disc (clusters)] status changed, 169.0 -> N/A
- hf_Longformer[dynamo-disc (clusters)] status changed, 172.0 -> N/A
- hf_Longformer[dynamo-disc (compiled)] status changed, 6130.0 -> N/A
- hf_T5[dynamo-disc (latency)] status changed, 32.34 -> OSError
- hf_T5[dynamo-disc (clusters)] status changed, 35.0 -> N/A
- hf_T5[dynamo-disc (compiled)] status changed, 1775.0 -> N/A
- hf_T5_base[dynamo-disc (latency)] status changed, 92.944 -> OSError
- hf_T5_base[dynamo-disc (clusters)] status changed, 65.0 -> N/A
- hf_T5_base[dynamo-disc (compiled)] status changed, 3431.0 -> N/A
- hf_T5_large[dynamo-blade (latency)] 67.117 -> 58.449, +12.9148%
- hf_T5_large[dynamo-disc (latency)] status changed, 61.988 -> OSError
- hf_T5_large[dynamo-disc (clusters)] status changed, 125.0 -> N/A
- hf_T5_large[dynamo-disc (compiled)] status changed, 6743.0 -> N/A
- maml_omniglot[blade (latency)] 0.274 -> 0.293, -6.9343%
- maml_omniglot[dynamo-disc (latency)] 0.645 -> 0.513, +20.4651%
- mnasnet1_0[dynamo-blade (latency)] 2.78 -> 2.61, +6.1151%
- mnasnet1_0[dynamo-disc (latency)] 4.175 -> 3.872, +7.2575%
- mobilenet_v3_large[dynamo-blade (latency)] 3.87 -> 3.563, +7.9328%
- phlippe_densenet[dynamo-blade (latency)] 4.085 -> 3.736, +8.5435%
- phlippe_densenet[dynamo-disc (latency)] 5.089 -> 4.767, +6.3274%
- pyhpc_equation_of_state[disc (latency)] 1.893 -> 2.01, -6.1807%
- pyhpc_isoneutral_mixing[blade (latency)] 7.07 -> 7.534, -6.5629%
- pyhpc_isoneutral_mixing[dynamo-blade (latency)] 6.201 -> 6.655, -7.3214%
- pyhpc_turbulent_kinetic_energy[blade (latency)] 5.746 -> 6.059, -5.4473%
- pyhpc_turbulent_kinetic_energy[dynamo-blade (latency)] 5.872 -> 6.222, -5.9605%
- pytorch_stargan[blade (latency)] 9.462 -> 26.297, -177.9222%
- resnet18[dynamo-blade (latency)] 0.993 -> 0.92, +7.3515%
- resnet18[dynamo-disc (latency)] 1.459 -> 1.368, +6.2371%
- resnext50_32x4d[dynamo-blade (latency)] 2.405 -> 2.283, +5.0728%
- squeezenet1_1[dynamo-blade (latency)] 0.977 -> 0.908, +7.0624%
- timm_efficientnet[dynamo-blade (latency)] 11.535 -> 10.802, +6.3546%
- timm_vision_transformer[dynamo-disc (latency)] 3.77 -> 3.424, +9.1777%
- timm_vovnet[dynamo-blade (latency)] 10.484 -> 9.135, +12.8672%
- timm_vovnet[dynamo-disc (latency)] 16.368 -> 15.307, +6.4822%
- yolov3[dynamo-disc (latency)] status changed, 27.479 -> RuntimeError
- yolov3[dynamo-disc (clusters)] status changed, 18.0 -> N/A
- yolov3[dynamo-disc (compiled)] status changed, 141.0 -> N/A

detail data can be seen in oss://bladedisc-ci/TorchBench/gpu/full/20230609-15
created by TorchBench CI automatically

zzp_miracle · Answer 1 · Thu Jun 29 2023 11:10:17 GMT+0800 (China Standard Time)

duplicated to #1180