Giters

Theano / Theano

Theano was a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It is being continued as PyTensor: www.github.com/pymc-devs/pytensor

Home Page:https://www.github.com/pymc-devs/pytensor

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

PushOutScanOutput crashes

rizar opened this issue · comments

commented

I see the following error message during optimization. I am using the latest Theano, device=cuda and optimizer=fast_run

2017-05-31 16:32:28,057: theano.gof.opt: ERROR: SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7faa1d96d190>
2017-05-31 16:32:28,057: theano.gof.opt: ERROR: Traceback:
2017-05-31 16:32:28,089: theano.gof.opt: ERROR: Traceback (most recent call last):
  File "/u/bahdanau/Dist/theano/theano/gof/opt.py", line 241, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/u/bahdanau/Dist/theano/theano/gof/opt.py", line 87, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/u/bahdanau/Dist/theano/theano/scan_module/scan_opt.py", line 685, in apply
    node = self.process_node(fgraph, node)
  File "/u/bahdanau/Dist/theano/theano/scan_module/scan_opt.py", line 745, in process_node
    node, args)
  File "/u/bahdanau/Dist/theano/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
    add_as_nitsots)
  File "/u/bahdanau/Dist/theano/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
    reason='scanOp_pushout_output')
  File "/u/bahdanau/Dist/theano/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/u/bahdanau/Dist/theano/theano/gof/toolbox.py", line 543, in replace_all_validate
    fgraph.validate()
  File "/u/bahdanau/Dist/theano/theano/gof/toolbox.py", line 434, in validate_
    ret = fgraph.execute_callbacks('validate')
  File "/u/bahdanau/Dist/theano/theano/gof/fg.py", line 594, in execute_callbacks
    fn(self, *args, **kwargs)
  File "/u/bahdanau/Dist/theano/theano/gof/toolbox.py", line 600, in validate
    raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
InconsistencyError: Trying to reintroduce a removed node
commented

To work around that (but it could slow down computation), can you try this Theano flag?

optimizer_excluding=scanOp_pushout_output

Do you have code that reproduce that? It will be hard to debug without that.