TENSORFLOW GPU 2.3的版本上TRAIN时总是出现ResizeNearestNeighbor错误
davidqing2000 opened this issue · comments
在TENSORFLOW GPU 2.3的版本上TRAIN时总是出现以下错误，如何处理？
Exception has occurred: InternalError
2 root error(s) found.
(0) Internal: Failed launching ResizeNearestNeighbor
[[{{node up_sampling2d/resize/ResizeNearestNeighbor}}]]
[[Func/training/Adam/gradients/gradients/batch_normalization_18/cond_grad/StatelessIf/then/_2092/input/_3921/_993]]
(1) Internal: Failed launching ResizeNearestNeighbor
[[{{node up_sampling2d/resize/ResizeNearestNeighbor}}]]
0 successful operations.
0 derived errors ignored.
File "C:\Work\py_dev\yolov4-tiny-tf2-master\train.py", line 262, in
callbacks=[logging, checkpoint, reduce_lr, early_stopping])
奇怪现象：当加入禁止GPU使用时，代码能正常工作，但使用GPU时，则出现上述错误。
如：增加以下
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
如果改变UpSampling2D去bilinear，可以运行了，但interpolation="nearest"则无法使用，不知是否哪里产生了BUG.
# Conv+UpSampling2D 13x13x256 -> 26x26x128
P5_upsample = compose(DarknetConv2D_BN_Leaky(128, (1,1)), UpSampling2D(2,interpolation='bilinear'))(P5)