UnknownError: CUDNN_STATUS_EXECUTION_FAILED Failed to set dropout descriptor

[robbine]

I got this strange error when trying to restore from previous trained checkpoints and continue training. It seems that cudnn conflicts with dropout descriptor.

UnknownError: CUDNN_STATUS_EXECUTION_FAILED Failed to set dropout descriptor
[[Node: tower_0/model_inputs/qst_embedding/CudnnRNN = CudnnRNN[T=DT_FLOAT, direction="bidirectional", dropout=0.4, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=87654321, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](tower_0/model_inputs/qst_embedding/dropout/mul, tower_0/model_inputs/qst_embedding/zeros, tower_0/model_inputs/qst_embedding/zeros, model_inputs/qst_embedding/lstm/train_buffer/read)]]
[[Node: tower_0/gradients/tower_0/ctx_qst_low_fusion/Reshape_7_grad/Reshape/_509 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_5180_tower_0/gradients/tower_0/ctx_qst_low_fusion/Reshape_7_grad/Reshape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op u'tower_0/model_inputs/qst_embedding/CudnnRNN', defined at:
File "/root/anaconda2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/root/anaconda2/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/root/anaconda2/lib/python2.7/site-packages/ipykernel/main.py", line 3, in
app.launch_new_instance()
File "/root/anaconda2/lib/python2.7/site-packages/traitlets/config/application.py", line 596, in launch_instance
app.start()
File "/root/anaconda2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 442, in start
ioloop.IOLoop.instance().start()
File "/root/anaconda2/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 162, in start
super(ZMQIOLoop, self).start()
File "/root/anaconda2/lib/python2.7/site-packages/tornado/ioloop.py", line 883, in start
handler_func(fd_obj, events)
File "/root/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "/root/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "/root/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "/root/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "/root/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "/root/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 276, in dispatcher
return self.dispatch_shell(stream, msg)
File "/root/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 228, in dispatch_shell
handler(stream, idents, msg)
File "/root/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 391, in execute_request
user_expressions, allow_stdin)
File "/root/anaconda2/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 199, in do_execute
shell.run_cell(code, store_history=store_history, silent=silent)
File "/root/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/root/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2828, in run_ast_nodes
if self.run_code(code, result):
File "/root/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
Trainer(options).train()
File "./qa/train/trainer.py", line 65, in train
compute_gradients=True, sess=self.session)
File "./qa/train/model_builder.py", line 23, in init
self._setup()
File "./qa/train/model_builder.py", line 83, in _setup
iterators)
File "./qa/train/model_builder.py", line 47, in _add_tower_and_compute_loss
tower.setup()
File "./qa/model/fusion_net.py", line 19, in setup
super(FusionNet, self).setup()
File "./qa/model/base_model.py", line 63, in setup
self.rnn_keep_prob)
File "./qa/model/input_util.py", line 178, in create_model_inputs
use_dropout, rnn_keep_prob)
File "./qa/model/input_util.py", line 92, in _add_char_embedding_inputs
sq_dataset, use_dropout, rnn_keep_prob)
File "./qa/model/input_util.py", line 85, in _run_cudnn_char_birnn
use_dropout) # size = [batch_size * N, 2, rnn_size]
File "./qa/model/cudnn_lstm_wrapper.py", line 121, in run_cudnn_lstm_and_return_hidden_outputs
initial_state_c=initial_state_c)
File "./qa/model/cudnn_lstm_wrapper.py", line 85, in run_cudnn_lstm
is_training=True)
File "/root/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 1443, in call
input_data, input_h, input_c, params, is_training=is_training)
File "/root/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 1334, in call
seed=self._seed)
File "/root/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 823, in _cudnn_rnn
name=name)
File "/root/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/cudnn_rnn/ops/gen_cudnn_rnn_ops.py", line 105, in cudnn_rnn
is_training=is_training, name=name)
File "/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

UnknownError (see above for traceback): CUDNN_STATUS_EXECUTION_FAILED Failed to set dropout descriptor
[[Node: tower_0/model_inputs/qst_embedding/CudnnRNN = CudnnRNN[T=DT_FLOAT, direction="bidirectional", dropout=0.4, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=87654321, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](tower_0/model_inputs/qst_embedding/dropout/mul, tower_0/model_inputs/qst_embedding/zeros, tower_0/model_inputs/qst_embedding/zeros, model_inputs/qst_embedding/lstm/train_buffer/read)]]
[[Node: tower_0/gradients/tower_0/ctx_qst_low_fusion/Reshape_7_grad/Reshape/_509 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_5180_tower_0/gradients/tower_0/ctx_qst_low_fusion/Reshape_7_grad/Reshape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

[obryanlouis]

I have a question for you - are you using the same cuDNN version when you restore the model? If not, the code may not handle it well since it's not using the "canonical" format to save cuDNN LSTM weights.

However, assuming that you are on the same version between runs, I normally just see this when I run out of GPU memory; if this is the case, then try reducing the batch size, or closing other programs using GPU memory (restarting the computer can help). See this thread with a similar error: pytorch/pytorch#953

[robbine]

I set batch size to 4 and it's running again. Thanks!