TestMnistConv.test_softmax fails

Question

TestMnistConv.test_softmax fails

Aetf opened this issue 7 years ago · comments

Aetf commented 7 years ago

TestMnistConv.test_softmax crashes in closeSession in tensorflow.

Steps to reproduce

launch executor
export TF_CPP_MIN_VLOG_LEVEL=0. This is important. With TF_CPP_MIN_VLOG_LEVEL=3 the test passes
python test_mnist_tf.py TestMnistConv.test_softmax

Expected

Test passes

Actual

Multiple crashes in tensorflow.

Crashes in tensorflow when handling closeSession: throws std::system_error (Invalid argument)
Segfault during Process, probably due to garbage value in tagged_node passed in.

Aetf · Answer 1 · Thu Jul 06 2017 02:40:19 GMT+0800 (China Standard Time)

Attached log. ~~Shows a crash during closeSession~~
crash-close-sess.tar.gz

It is actually a crash when run the network on CPU in TF.

Aetf · Answer 2 · Thu Jul 06 2017 04:37:37 GMT+0800 (China Standard Time)

Doing a bisect shows that the crash was introduced in SymbioticLab/tensorflow-salus@a927079b4. Fixed by removing those logs, which were used for debugging thread pool hanging issue, and is no longer needed.

Fixed in SymbioticLab/tensorflow-salus@94f320232