Failures on unittests for otx.core.ov modules
yunchu opened this issue · comments
Describe the bug
After upgrading openvino version to 2023.1, downloaded omz models were compressed with option of mo "--compress_to_fp16 true" as default. this leads some unittests failure.
ref. openvino 2023.1 release note
Steps to Reproduce
- delete cached omz model file if it exists
$ rm -rf ~/.cache/otx/omz- run unittests
$ tox -vvv -e unittest-all-py310-pt1 -- tests/unit/core/ov/graph/test_ov_graph_utils.py::test_handle_reshape- TypeError will be occurred
========================================================================== short test summary info ==========================================================================
FAILED tests/unit/core/ov/graph/test_ov_graph_utils.py::test_handle_reshape - TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uin...
Environment:
- OS: Ubuntu 20.04
- Framework version: Pytorch
- Python version: 3.9
- OpenVINO version: 2023.1
- CUDA/cuDNN version: 11.7
- GPU model and memory:
some of the CI runners have uncompressed model files in their cache directory so that would not be resulted as failure. but on the un-cached machine, TC failures will be occurred due to the compressed model is not suitable to some current unittests design.
I think that this test would be better to be moved to "intergation" test. This is because this test can be affected by the external components: 1) OpenVINO Model Optimizer and 2) Downloaded IR file. In addition, the integration test should not be stateful (affected by the local caching directory like this).
@yunchu thanks for the investigation! Could you delete the cache where applicable?
@vinnamkim indeed, current design doesn't correspond to the unit tests conception. Are there any plans to add this OV graph parsing functionality to OTX 2.0? If yes, we could think about refactoring in advance
@yunchu thanks for the investigation! Could you delete the cache where applicable?
@vinnamkim indeed, current design doesn't correspond to the unit tests conception. Are there any plans to add this OV graph parsing functionality to OTX 2.0? If yes, we could think about refactoring in advance
@sovrasov actually deleting cache would not resolve this issue. we have bumped up the version of openvino to 2023.1 so the downloaded and converted (via mo) model will be compressed with fp16 by the default option to the mo. it leads current unittest failures.
FYI, i've manually pushed uncompressed version of models to the CI runner's cache now. so, i think related TCs would be passed from now on.
@vinnamkim indeed, current design doesn't correspond to the unit tests conception. Are there any plans to add this OV graph parsing functionality to OTX 2.0? If yes, we could think about refactoring in advance
Yea, I'm planning to move all test functions which require external components explicitly like this to the integration test.
@yunchu thanks for the investigation! Could you delete the cache where applicable?
@vinnamkim indeed, current design doesn't correspond to the unit tests conception. Are there any plans to add this OV graph parsing functionality to OTX 2.0? If yes, we could think about refactoring in advance@sovrasov actually deleting cache would not resolve this issue. we have bumped up the version of openvino to 2023.1 so the downloaded and converted (via mo) model will be compressed with fp16 by the default option to the mo. it leads current unittest failures.
I asked this in the context of merging #2676
But keeping this cache is also an option, although it's implicit and should be maintained over time.
Closed, since it's related to past 1.6 release