loss becomes nan
GeorgeShimakawa opened this issue · comments
When I switched cpu based Keras to PaidML based Keras, loss values became "nan" while running Istm.py. It was not the case before. Would you guess what the reason is ?
Pardon for my ambiguous question.
I changed backend to an AMD GPU "PlaidML" by adding a line (os.environ["KERAS_BACKEND"] = "plaidml.keras.backend"). Platform is iMac Late2015, OSX-Mojave, Python 3.6.5
After parsing about 60 midi files, first loss was indicated as 'nan' as follows:
.
.
.
Parsing midi_songs/sonat-5.mid
Parsing midi_songs/352.mid
Parsing midi_songs/384.mid
Parsing midi_songs/5112.mid
Parsing midi_songs/sonat-4.mid
INFO:plaidml:Opening device "metal_amd_radeon_r9_m390.0"
Epoch 1/200
INFO:plaidml:Analyzing Ops: 2104 of 29742 operations complete
INFO:plaidml:Analyzing Ops: 5776 of 29742 operations complete
INFO:plaidml:Analyzing Ops: 9603 of 29742 operations complete
INFO:plaidml:Analyzing Ops: 14197 of 29742 operations complete
INFO:plaidml:Analyzing Ops: 19025 of 29742 operations complete
INFO:plaidml:Analyzing Ops: 24197 of 29742 operations complete
INFO:plaidml:Analyzing Ops: 27148 of 29742 operations complete
720/43019 [..............................] - ETA: 1:39:10 - loss: nan
Even when I continued further the the loss continues to be nan.
OpenCL version is 1.2 and the gpu is AMD Radeon R9 M390. CPU is Intel Core i5.
PlaidML is 0.3.5. Memory is 16GB.