Падение бенчмаркинга OpenVINO в Docker
FenixFly opened this issue · comments
После последних коммитов началось падение бенчмаркинга, когда производится замер моделей в Docker.
Лог запуска и ошибки на машине i7. Падает 6й по счету тест
Start inference test on model: alexnet
Command line is: python3 /tmp/dl-benchmark/src/inference/inference_async_mode.py -m /mnt/models/public/alexnet/FP32/alexnet.xml -w /mnt/models/public/alexnet/FP32/alexnet.bin -i /tmp/itlab-vision-dl-benchmark-data/Datasets/ImageNET/ -b 8 -d CPU -ni 1000 --raw_output true
�[33;20m[ WARNING ] Inference test on model: alexnet was ended with error. Process logs:�[0m
�[31;20m[ ERROR ] Inference failed with exception: sequence item 7: expected str instance, NoneType found�[0m
Traceback (most recent call last):
File "inference_benchmark.py", line 80, in inference_benchmark
test_process.execute()
File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/processes.py", line 59, in execute
self.__save_failed_test_log()
File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/processes.py", line 153, in __save_failed_test_log
log_filename = self.__make_log_filename()
File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/processes.py", line 174, in __make_log_filename
filename = '_'.join(test_settings)
TypeError: sequence item 7: expected str instance, NoneType found
Saving test result in file
�[31;20m[ ERROR ] Uncaught exception: 'NoneType' object has no attribute 'get_status'. Traceback: [' File "inference_benchmark.py", line 114, in <module>\n inference_status = inference_benchmark(args.executor_type, test_list,\n', ' File "inference_benchmark.py", line 92, in inference_benchmark\n output_handler.add_row_to_table(process_executor, test, test_process)\n', ' File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/output.py", line 57, in add_row_to_table\n report_row = self.__create_table_row(executor, test, process)\n', ' File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/output.py", line 36, in __create_table_row\n report = test.get_report(process=process)\n', ' File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/openvino/openvino_test.py", line 7, in get_report\n if process.get_status() == 0 and not tensors_num:\n']�[0m
Лог запуска и ошибки на машине i3. Падает 79й по счету тест
Start inference test on model: inception-resnet-v2-tf
Command line is: python3 /tmp/dl-benchmark/src/inference/inference_sync_mode.py -m /mnt/models/public/inception-resnet-v2/FP32/inception-resnet-v2-tf.xml -w /mnt/models/public/inception-resnet-v2/FP32/inception-resnet-v2-tf.bin -i /tmp/itlab-vision-dl-benchmark-data/Datasets/ImageNET/ -b 1 -d CPU -ni 1000 --raw_output true
�[33;20m[ WARNING ] Inference test on model: inception-resnet-v2-tf was ended with error. Process logs:�[0m
�[31;20m[ ERROR ] Inference failed with exception: sequence item 7: expected str instance, NoneType found�[0m
Traceback (most recent call last):
File "inference_benchmark.py", line 80, in inference_benchmark
test_process.execute()
File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/processes.py", line 59, in execute
self.__save_failed_test_log()
File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/processes.py", line 153, in __save_failed_test_log
log_filename = self.__make_log_filename()
File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/processes.py", line 174, in __make_log_filename
filename = '_'.join(test_settings)
TypeError: sequence item 7: expected str instance, NoneType found
Saving test result in file
�[31;20m[ ERROR ] Uncaught exception: 'NoneType' object has no attribute 'get_status'. Traceback: [' File "inference_benchmark.py", line 114, in <module>\n inference_status = inference_benchmark(args.executor_type, test_list,\n', ' File "inference_benchmark.py", line 92, in inference_benchmark\n output_handler.add_row_to_table(process_executor, test, test_process)\n', ' File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/output.py", line 57, in add_row_to_table\n report_row = self.__create_table_row(executor, test, process)\n', ' File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/output.py", line 36, in __create_table_row\n report = test.get_report(process=process)\n', ' File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/openvino/openvino_test.py", line 7, in get_report\n if process.get_status() == 0 and not tensors_num:\n']�[0m
Лог запуска и ошибки на машине tower
Command line is: python3 /tmp/dl-benchmark/src/inference/inference_sync_mode.py -m /mnt/models/public/inception-resnet-v2/FP32/inception-resnet-v2-tf.xml -w /mnt/models/public/inception-resnet-v2/FP32/inception-resnet-v2-tf.bin -i /tmp/itlab-vision-dl-benchmark-data/Datasets/ImageNET/ -b 1 -d CPU -ni 1000 --raw_output true
�[33;20m[ WARNING ] Inference test on model: inception-resnet-v2-tf was ended with error. Process logs:�[0m
�[31;20m[ ERROR ] Inference failed with exception: sequence item 7: expected str instance, NoneType found�[0m
Traceback (most recent call last):
File "inference_benchmark.py", line 80, in inference_benchmark
test_process.execute()
File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/processes.py", line 59, in execute
self.__save_failed_test_log()
File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/processes.py", line 153, in __save_failed_test_log
log_filename = self.__make_log_filename()
File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/processes.py", line 174, in __make_log_filename
filename = '_'.join(test_settings)
TypeError: sequence item 7: expected str instance, NoneType found
Saving test result in file
�[31;20m[ ERROR ] Uncaught exception: 'NoneType' object has no attribute 'get_status'. Traceback: [' File "inference_benchmark.py", line 114, in <module>\n inference_status = inference_benchmark(args.executor_type, test_list,\n', ' File "inference_benchmark.py", line 92, in inference_benchmark\n output_handler.add_row_to_table(process_executor, test, test_process)\n', ' File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/output.py", line 57, in add_row_to_table\n report_row = self.__create_table_row(executor, test, process)\n', ' File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/output.py", line 36, in __create_table_row\n report = test.get_report(process=process)\n', ' File "/home/itmm/dli-jenkins-worker/dl-benchmark/src/benchmark/frameworks/openvino/openvino_test.py", line 7, in get_report\n if process.get_status() == 0 and not tensors_num:\n']�[0m
@FenixFly, нужен конфигурационный файл бенчмарка, на котором происходит падение.
Конфиг падающего теста с i7:
<Test>
<Model>
<Task>classification</Task>
<Name>alexnet</Name>
<Precision>FP32</Precision>
<SourceFramework>Caffe</SourceFramework>
<ModelPath>/mnt/models/public/alexnet/FP32/alexnet.xml</ModelPath>
<WeightsPath>/mnt/models/public/alexnet/FP32/alexnet.bin</WeightsPath>
</Model>
<Dataset>
<Name>ImageNET</Name>
<Path>/tmp/itlab-vision-dl-benchmark-data/Datasets/ImageNET/</Path>
</Dataset>
<FrameworkIndependent>
<InferenceFramework>OpenVINO DLDT</InferenceFramework>
<BatchSize>8</BatchSize>
<Device>CPU</Device>
<IterationCount>1000</IterationCount>
<TestTimeLimit>180</TestTimeLimit>
</FrameworkIndependent>
<FrameworkDependent>
<Mode>async</Mode>
<Extension></Extension>
<AsyncRequestCount></AsyncRequestCount>
<ThreadCount></ThreadCount>
<StreamCount></StreamCount>
</FrameworkDependent>
</Test>
Конфиг падающего теста с i3
<Test>
<Model>
<Task>classification</Task>
<Name>inception-resnet-v2-tf</Name>
<Precision>FP32</Precision>
<SourceFramework>TensorFlow</SourceFramework>
<ModelPath>/mnt/models/public/inception-resnet-v2/FP32/inception-resnet-v2-tf.xml</ModelPath>
<WeightsPath>/mnt/models/public/inception-resnet-v2/FP32/inception-resnet-v2-tf.bin</WeightsPath>
</Model>
<Dataset>
<Name>ImageNET</Name>
<Path>/tmp/itlab-vision-dl-benchmark-data/Datasets/ImageNET/</Path>
</Dataset>
<FrameworkIndependent>
<InferenceFramework>OpenVINO DLDT</InferenceFramework>
<BatchSize>1</BatchSize>
<Device>CPU</Device>
<IterationCount>1000</IterationCount>
<TestTimeLimit>180</TestTimeLimit>
</FrameworkIndependent>
<FrameworkDependent>
<Mode>sync</Mode>
<Extension></Extension>
<AsyncRequestCount></AsyncRequestCount>
<ThreadCount></ThreadCount>
<StreamCount></StreamCount>
</FrameworkDependent>
</Test>
@maslovaz спасибо, помогло!