RuntimeError: result type ComplexFloat can't be cast to the desired output type Float
yunchu opened this issue · comments
Describe the bug
tested on branch releases/1.5.0
failures on regression test for semantic segmentation
- train type: class incr
- template
- SegNext-B, SegNext-t, SegNext-s
Steps to Reproduce
- copy ci datasets to local
- run regression test with following command
CI_DATA_ROOT=/mnt/hdd1/data/ci_datasets/ tox -vvv -e tests-seg-py310-pt1 -- tests/regression/semantic_segmentation/test_segmentation.py::TestRegressionSegmentation::test_otx_train[Custom_Semantic_Segmentation_SegNext_B] tests/regression/semantic_segmentation/test_segmentation.py::TestRegressionSegmentation::test_otx_train_cls_incr[Custom_Semantic_Segmentation_SegNext_B]
Environment:
- OS:
- Framework version:
- Python version:
- OpenVINO version:
- CUDA/cuDNN version:
- GPU model and memory:
@yunchu , I investigated this issue and found some weird behavior in a test environment. This issue happens because of in train_cls_incr we change datasets, but the number of iterations is much lower than expected, so, runner.current_iter < runner,max_iters, and there are some issues in the scheduler emerge.
Can it be some possible shift in two testing environments (otx_train and train_cls_incr) leading to broken information about proper dataset size/number of epochs?
BTW,
This problem is not reproducible outside of tests. I tried the class incremental approach with the same data and everything is fine. The number of iterations is what it should be without any errors.
@kprokofi I guess you can check the regression test on the repo. I've just checked that was happened on 'develop' branch as well.
AFAIK, the current regression test run only 10 epoch for each training. do you think it could make any issues?