Any ideas on: InvalidArgumentError: Assign requires shapes of both tensors to match.
jacktionman opened this issue · comments
Have gone through this great tutorial (thank you for it!) but am hitting an issue I haven't seen.
On Gunicorn/Flask running on my Heroku server, I get an error saying my tensor shapes don't match:
lhs shape= [52293,48] rhs shape= [52293,112]
Not sure what to do with this, as I trained them using the code as expected in the tutorial.
Do you think this is a training error or a Flask implementation issue? I did update the Py2 code to run on Py3, may have introduced an error. But that was mainly adding parenthesis and cleaning up tabs/spacing - I'm not sure why that would cause different tensor shapes?
Any suggestions would be greatly appreciated.
Log dump below.
2018-05-12T00:36:16.577701+00:00 heroku[web.1]: State changed from crashed to starting
2018-05-12T00:36:51.453212+00:00 heroku[web.1]: Starting process with command `gunicorn app:app`
2018-05-12T00:36:54.374191+00:00 app[web.1]: [2018-05-12 00:36:54 +0000] [4] [INFO] Starting gunicorn 19.8.1
2018-05-12T00:36:54.375117+00:00 app[web.1]: [2018-05-12 00:36:54 +0000] [4] [INFO] Listening at: http://0.0.0.0:45940 (4)
2018-05-12T00:36:54.375270+00:00 app[web.1]: [2018-05-12 00:36:54 +0000] [4] [INFO] Using worker: sync
2018-05-12T00:36:54.380861+00:00 app[web.1]: [2018-05-12 00:36:54 +0000] [8] [INFO] Booting worker with pid: 8
2018-05-12T00:36:54.444401+00:00 app[web.1]: [2018-05-12 00:36:54 +0000] [9] [INFO] Booting worker with pid: 9
2018-05-12T00:36:55.354932+00:00 heroku[web.1]: State changed from starting to up
2018-05-12T00:37:22.145542+00:00 app[web.1]: [2018-05-12 00:37:22 +0000] [8] [ERROR] Exception in worker process
2018-05-12T00:37:22.145573+00:00 app[web.1]: Traceback (most recent call last):
2018-05-12T00:37:22.145575+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
2018-05-12T00:37:22.145577+00:00 app[web.1]: return fn(*args)
2018-05-12T00:37:22.145579+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
2018-05-12T00:37:22.145581+00:00 app[web.1]: options, feed_dict, fetch_list, target_list, run_metadata)
2018-05-12T00:37:22.145582+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
2018-05-12T00:37:22.145584+00:00 app[web.1]: run_metadata)
2018-05-12T00:37:22.145586+00:00 app[web.1]: tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [52293,48] rhs shape= [52293,112]
2018-05-12T00:37:22.145588+00:00 app[web.1]: [[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding, save/RestoreV2:7)]]
2018-05-12T00:37:22.145590+00:00 app[web.1]:
2018-05-12T00:37:22.145592+00:00 app[web.1]: During handling of the above exception, another exception occurred:
2018-05-12T00:37:22.145593+00:00 app[web.1]:
2018-05-12T00:37:22.145595+00:00 app[web.1]: Traceback (most recent call last):
2018-05-12T00:37:22.145597+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2018-05-12T00:37:22.145598+00:00 app[web.1]: worker.init_process()
2018-05-12T00:37:22.145600+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2018-05-12T00:37:22.145601+00:00 app[web.1]: self.load_wsgi()
2018-05-12T00:37:22.145603+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2018-05-12T00:37:22.145604+00:00 app[web.1]: self.wsgi = self.app.wsgi()
2018-05-12T00:37:22.145606+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2018-05-12T00:37:22.145607+00:00 app[web.1]: self.callable = self.load()
2018-05-12T00:37:22.145609+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2018-05-12T00:37:22.145611+00:00 app[web.1]: return self.load_wsgiapp()
2018-05-12T00:37:22.145612+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2018-05-12T00:37:22.145613+00:00 app[web.1]: return util.import_app(self.app_uri)
2018-05-12T00:37:22.145615+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2018-05-12T00:37:22.145617+00:00 app[web.1]: __import__(module)
2018-05-12T00:37:22.145618+00:00 app[web.1]: File "/app/app.py", line 40, in <module>
2018-05-12T00:37:22.145620+00:00 app[web.1]: saver.restore(sess, tf.train.latest_checkpoint('models'))
2018-05-12T00:37:22.145621+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1802, in restore
2018-05-12T00:37:22.145623+00:00 app[web.1]: {self.saver_def.filename_tensor_name: save_path})
2018-05-12T00:37:22.145624+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
2018-05-12T00:37:22.145626+00:00 app[web.1]: run_metadata_ptr)
2018-05-12T00:37:22.145627+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
2018-05-12T00:37:22.145629+00:00 app[web.1]: feed_dict_tensor, options, run_metadata)
2018-05-12T00:37:22.145630+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
2018-05-12T00:37:22.145632+00:00 app[web.1]: run_metadata)
2018-05-12T00:37:22.145633+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
2018-05-12T00:37:22.145635+00:00 app[web.1]: raise type(e)(node_def, op, message)
2018-05-12T00:37:22.145636+00:00 app[web.1]: tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [52293,48] rhs shape= [52293,112]
2018-05-12T00:37:22.145638+00:00 app[web.1]: [[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding, save/RestoreV2:7)]]
2018-05-12T00:37:22.145640+00:00 app[web.1]:
2018-05-12T00:37:22.145641+00:00 app[web.1]: Caused by op 'save/Assign_7', defined at:
2018-05-12T00:37:22.145643+00:00 app[web.1]: File "/app/.heroku/python/bin/gunicorn", line 11, in <module>
2018-05-12T00:37:22.145644+00:00 app[web.1]: sys.exit(run())
2018-05-12T00:37:22.145646+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 61, in run
2018-05-12T00:37:22.145647+00:00 app[web.1]: WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
2018-05-12T00:37:22.145649+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 223, in run
2018-05-12T00:37:22.145650+00:00 app[web.1]: super(Application, self).run()
2018-05-12T00:37:22.145652+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 72, in run
2018-05-12T00:37:22.145653+00:00 app[web.1]: Arbiter(self).run()
2018-05-12T00:37:22.145655+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 203, in run
2018-05-12T00:37:22.145656+00:00 app[web.1]: self.manage_workers()
2018-05-12T00:37:22.145668+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
2018-05-12T00:37:22.145669+00:00 app[web.1]: self.spawn_workers()
2018-05-12T00:37:22.145671+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
2018-05-12T00:37:22.145673+00:00 app[web.1]: self.spawn_worker()
2018-05-12T00:37:22.145674+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2018-05-12T00:37:22.145676+00:00 app[web.1]: worker.init_process()
2018-05-12T00:37:22.145677+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2018-05-12T00:37:22.145679+00:00 app[web.1]: self.load_wsgi()
2018-05-12T00:37:22.145680+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2018-05-12T00:37:22.145681+00:00 app[web.1]: self.wsgi = self.app.wsgi()
2018-05-12T00:37:22.145683+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2018-05-12T00:37:22.145684+00:00 app[web.1]: self.callable = self.load()
2018-05-12T00:37:22.145686+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2018-05-12T00:37:22.145687+00:00 app[web.1]: return self.load_wsgiapp()
2018-05-12T00:37:22.145689+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2018-05-12T00:37:22.145690+00:00 app[web.1]: return util.import_app(self.app_uri)
2018-05-12T00:37:22.145692+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2018-05-12T00:37:22.145695+00:00 app[web.1]: __import__(module)
2018-05-12T00:37:22.145696+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2018-05-12T00:37:22.145698+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2018-05-12T00:37:22.145699+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2018-05-12T00:37:22.145701+00:00 app[web.1]: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
2018-05-12T00:37:22.145702+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2018-05-12T00:37:22.145704+00:00 app[web.1]: File "/app/app.py", line 39, in <module>
2018-05-12T00:37:22.145705+00:00 app[web.1]: saver = tf.train.Saver()
2018-05-12T00:37:22.145707+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1338, in __init__
2018-05-12T00:37:22.145709+00:00 app[web.1]: self.build()
2018-05-12T00:37:22.145710+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1347, in build
2018-05-12T00:37:22.145712+00:00 app[web.1]: self._build(self._filename, build_save=True, build_restore=True)
2018-05-12T00:37:22.145713+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1384, in _build
2018-05-12T00:37:22.145715+00:00 app[web.1]: build_save=build_save, build_restore=build_restore)
2018-05-12T00:37:22.145716+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 835, in _build_internal
2018-05-12T00:37:22.145724+00:00 app[web.1]: restore_sequentially, reshape)
2018-05-12T00:37:22.145726+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 494, in _AddRestoreOps
2018-05-12T00:37:22.145728+00:00 app[web.1]: assign_ops.append(saveable.restore(saveable_tensors, shapes))
2018-05-12T00:37:22.145729+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 185, in restore
2018-05-12T00:37:22.145731+00:00 app[web.1]: self.op.get_shape().is_fully_defined())
2018-05-12T00:37:22.145733+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 283, in assign
2018-05-12T00:37:22.145734+00:00 app[web.1]: validate_shape=validate_shape)
2018-05-12T00:37:22.145736+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
2018-05-12T00:37:22.145737+00:00 app[web.1]: use_locking=use_locking, name=name)
2018-05-12T00:37:22.145739+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
2018-05-12T00:37:22.145741+00:00 app[web.1]: op_def=op_def)
2018-05-12T00:37:22.145742+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
2018-05-12T00:37:22.145744+00:00 app[web.1]: op_def=op_def)
2018-05-12T00:37:22.145745+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
2018-05-12T00:37:22.145747+00:00 app[web.1]: self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
2018-05-12T00:37:22.145748+00:00 app[web.1]:
2018-05-12T00:37:22.145750+00:00 app[web.1]: InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [52293,48] rhs shape= [52293,112]
2018-05-12T00:37:22.145752+00:00 app[web.1]: [[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding, save/RestoreV2:7)]]
2018-05-12T00:37:22.145753+00:00 app[web.1]:
2018-05-12T00:37:22.146397+00:00 app[web.1]: [2018-05-12 00:37:22 +0000] [8] [INFO] Worker exiting (pid: 8)
2018-05-12T00:37:24.191837+00:00 app[web.1]: [2018-05-12 00:37:24 +0000] [9] [ERROR] Exception in worker process
2018-05-12T00:37:24.191861+00:00 app[web.1]: Traceback (most recent call last):
2018-05-12T00:37:24.191863+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
2018-05-12T00:37:24.191866+00:00 app[web.1]: return fn(*args)
2018-05-12T00:37:24.191867+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
2018-05-12T00:37:24.191869+00:00 app[web.1]: options, feed_dict, fetch_list, target_list, run_metadata)
2018-05-12T00:37:24.191871+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
2018-05-12T00:37:24.191873+00:00 app[web.1]: run_metadata)
2018-05-12T00:37:24.191875+00:00 app[web.1]: tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [52293,48] rhs shape= [52293,112]
2018-05-12T00:37:24.191877+00:00 app[web.1]: [[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding, save/RestoreV2:7)]]
2018-05-12T00:37:24.191879+00:00 app[web.1]:
2018-05-12T00:37:24.191881+00:00 app[web.1]: During handling of the above exception, another exception occurred:
2018-05-12T00:37:24.191883+00:00 app[web.1]:
2018-05-12T00:37:24.191885+00:00 app[web.1]: Traceback (most recent call last):
2018-05-12T00:37:24.191887+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2018-05-12T00:37:24.191889+00:00 app[web.1]: worker.init_process()
2018-05-12T00:37:24.191890+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2018-05-12T00:37:24.191892+00:00 app[web.1]: self.load_wsgi()
2018-05-12T00:37:24.191894+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2018-05-12T00:37:24.191895+00:00 app[web.1]: self.wsgi = self.app.wsgi()
2018-05-12T00:37:24.191897+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2018-05-12T00:37:24.191899+00:00 app[web.1]: self.callable = self.load()
2018-05-12T00:37:24.191901+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2018-05-12T00:37:24.191902+00:00 app[web.1]: return self.load_wsgiapp()
2018-05-12T00:37:24.191904+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2018-05-12T00:37:24.191908+00:00 app[web.1]: return util.import_app(self.app_uri)
2018-05-12T00:37:24.191910+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2018-05-12T00:37:24.191912+00:00 app[web.1]: __import__(module)
2018-05-12T00:37:24.191913+00:00 app[web.1]: File "/app/app.py", line 40, in <module>
2018-05-12T00:37:24.191915+00:00 app[web.1]: saver.restore(sess, tf.train.latest_checkpoint('models'))
2018-05-12T00:37:24.191917+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1802, in restore
2018-05-12T00:37:24.191918+00:00 app[web.1]: {self.saver_def.filename_tensor_name: save_path})
2018-05-12T00:37:24.191920+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
2018-05-12T00:37:24.191922+00:00 app[web.1]: run_metadata_ptr)
2018-05-12T00:37:24.191923+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
2018-05-12T00:37:24.191925+00:00 app[web.1]: feed_dict_tensor, options, run_metadata)
2018-05-12T00:37:24.191927+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
2018-05-12T00:37:24.191928+00:00 app[web.1]: run_metadata)
2018-05-12T00:37:24.191930+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
2018-05-12T00:37:24.191932+00:00 app[web.1]: raise type(e)(node_def, op, message)
2018-05-12T00:37:24.191933+00:00 app[web.1]: tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [52293,48] rhs shape= [52293,112]
2018-05-12T00:37:24.191935+00:00 app[web.1]: [[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding, save/RestoreV2:7)]]
2018-05-12T00:37:24.191937+00:00 app[web.1]:
2018-05-12T00:37:24.191939+00:00 app[web.1]: Caused by op 'save/Assign_7', defined at:
2018-05-12T00:37:24.191940+00:00 app[web.1]: File "/app/.heroku/python/bin/gunicorn", line 11, in <module>
2018-05-12T00:37:24.191942+00:00 app[web.1]: sys.exit(run())
2018-05-12T00:37:24.191943+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 61, in run
2018-05-12T00:37:24.191945+00:00 app[web.1]: WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
2018-05-12T00:37:24.191947+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 223, in run
2018-05-12T00:37:24.191949+00:00 app[web.1]: super(Application, self).run()
2018-05-12T00:37:24.191950+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 72, in run
2018-05-12T00:37:24.191952+00:00 app[web.1]: Arbiter(self).run()
2018-05-12T00:37:24.191954+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 203, in run
2018-05-12T00:37:24.191956+00:00 app[web.1]: self.manage_workers()
2018-05-12T00:37:24.191967+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
2018-05-12T00:37:24.191969+00:00 app[web.1]: self.spawn_workers()
2018-05-12T00:37:24.191970+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
2018-05-12T00:37:24.191972+00:00 app[web.1]: self.spawn_worker()
2018-05-12T00:37:24.191974+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2018-05-12T00:37:24.191975+00:00 app[web.1]: worker.init_process()
2018-05-12T00:37:24.191977+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2018-05-12T00:37:24.191979+00:00 app[web.1]: self.load_wsgi()
2018-05-12T00:37:24.191980+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2018-05-12T00:37:24.191982+00:00 app[web.1]: self.wsgi = self.app.wsgi()
2018-05-12T00:37:24.191984+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2018-05-12T00:37:24.191985+00:00 app[web.1]: self.callable = self.load()
2018-05-12T00:37:24.191987+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2018-05-12T00:37:24.191989+00:00 app[web.1]: return self.load_wsgiapp()
2018-05-12T00:37:24.191990+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2018-05-12T00:37:24.191992+00:00 app[web.1]: return util.import_app(self.app_uri)
2018-05-12T00:37:24.191994+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2018-05-12T00:37:24.191995+00:00 app[web.1]: __import__(module)
2018-05-12T00:37:24.191997+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2018-05-12T00:37:24.191999+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2018-05-12T00:37:24.192001+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2018-05-12T00:37:24.192002+00:00 app[web.1]: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
2018-05-12T00:37:24.192004+00:00 app[web.1]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2018-05-12T00:37:24.192005+00:00 app[web.1]: File "/app/app.py", line 39, in <module>
2018-05-12T00:37:24.192007+00:00 app[web.1]: saver = tf.train.Saver()
2018-05-12T00:37:24.192009+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1338, in __init__
2018-05-12T00:37:24.192011+00:00 app[web.1]: self.build()
2018-05-12T00:37:24.192013+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1347, in build
2018-05-12T00:37:24.192014+00:00 app[web.1]: self._build(self._filename, build_save=True, build_restore=True)
2018-05-12T00:37:24.192016+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1384, in _build
2018-05-12T00:37:24.192018+00:00 app[web.1]: build_save=build_save, build_restore=build_restore)
2018-05-12T00:37:24.192019+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 835, in _build_internal
2018-05-12T00:37:24.192021+00:00 app[web.1]: restore_sequentially, reshape)
2018-05-12T00:37:24.192023+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 494, in _AddRestoreOps
2018-05-12T00:37:24.192025+00:00 app[web.1]: assign_ops.append(saveable.restore(saveable_tensors, shapes))
2018-05-12T00:37:24.192026+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 185, in restore
2018-05-12T00:37:24.192028+00:00 app[web.1]: self.op.get_shape().is_fully_defined())
2018-05-12T00:37:24.192030+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 283, in assign
2018-05-12T00:37:24.192031+00:00 app[web.1]: validate_shape=validate_shape)
2018-05-12T00:37:24.192033+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
2018-05-12T00:37:24.192034+00:00 app[web.1]: use_locking=use_locking, name=name)
2018-05-12T00:37:24.192037+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
2018-05-12T00:37:24.192038+00:00 app[web.1]: op_def=op_def)
2018-05-12T00:37:24.192040+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
2018-05-12T00:37:24.192042+00:00 app[web.1]: op_def=op_def)
2018-05-12T00:37:24.192043+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
2018-05-12T00:37:24.192045+00:00 app[web.1]: self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
2018-05-12T00:37:24.192047+00:00 app[web.1]:
2018-05-12T00:37:24.192049+00:00 app[web.1]: InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [52293,48] rhs shape= [52293,112]
2018-05-12T00:37:24.192050+00:00 app[web.1]: [[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](embedding_rnn_seq2seq/rnn/embedding_wrapper/embedding, save/RestoreV2:7)]]
2018-05-12T00:37:24.192052+00:00 app[web.1]:
2018-05-12T00:37:24.193093+00:00 app[web.1]: [2018-05-12 00:37:24 +0000] [9] [INFO] Worker exiting (pid: 9)
2018-05-12T00:37:24.811289+00:00 app[web.1]: Traceback (most recent call last):
2018-05-12T00:37:24.811360+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 210, in run
2018-05-12T00:37:24.811770+00:00 app[web.1]: self.sleep()
2018-05-12T00:37:24.811805+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 360, in sleep
2018-05-12T00:37:24.812141+00:00 app[web.1]: ready = select.select([self.PIPE[0]], [], [], 1.0)
2018-05-12T00:37:24.812178+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 245, in handle_chld
2018-05-12T00:37:24.812444+00:00 app[web.1]: self.reap_workers()
2018-05-12T00:37:24.812482+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
2018-05-12T00:37:24.812864+00:00 app[web.1]: raise HaltServer(reason, self.WORKER_BOOT_ERROR)
2018-05-12T00:37:24.812932+00:00 app[web.1]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
2018-05-12T00:37:24.812965+00:00 app[web.1]:
2018-05-12T00:37:24.812968+00:00 app[web.1]: During handling of the above exception, another exception occurred:
2018-05-12T00:37:24.812969+00:00 app[web.1]:
2018-05-12T00:37:24.813002+00:00 app[web.1]: Traceback (most recent call last):
2018-05-12T00:37:24.813032+00:00 app[web.1]: File "/app/.heroku/python/bin/gunicorn", line 11, in <module>
2018-05-12T00:37:24.813199+00:00 app[web.1]: sys.exit(run())
2018-05-12T00:37:24.813230+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 61, in run
2018-05-12T00:37:24.813414+00:00 app[web.1]: WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
2018-05-12T00:37:24.813450+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 223, in run
2018-05-12T00:37:24.813710+00:00 app[web.1]: super(Application, self).run()
2018-05-12T00:37:24.813746+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 72, in run
2018-05-12T00:37:24.813937+00:00 app[web.1]: Arbiter(self).run()
2018-05-12T00:37:24.813970+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 232, in run
2018-05-12T00:37:24.814214+00:00 app[web.1]: self.halt(reason=inst.reason, exit_status=inst.exit_status)
2018-05-12T00:37:24.814275+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 345, in halt
2018-05-12T00:37:24.814587+00:00 app[web.1]: self.stop()
2018-05-12T00:37:24.814618+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 393, in stop
2018-05-12T00:37:24.814948+00:00 app[web.1]: time.sleep(0.1)
2018-05-12T00:37:24.814986+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 245, in handle_chld
2018-05-12T00:37:24.815218+00:00 app[web.1]: self.reap_workers()
2018-05-12T00:37:24.815253+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
2018-05-12T00:37:24.815679+00:00 app[web.1]: raise HaltServer(reason, self.WORKER_BOOT_ERROR)
2018-05-12T00:37:24.815718+00:00 app[web.1]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
2018-05-12T00:37:24.926274+00:00 heroku[web.1]: Process exited with status 1
2018-05-12T00:37:24.993939+00:00 heroku[web.1]: State changed from up to crashed
That error normally means that your the current model is different from the pre-trained model. Double check to make sure that the shapes of all your variables are the same as the graph you saved, same number of LSTM units, etc.
Ah, you're totally right!
I was so worried that the conversion from Py2 to 3 had messed something up I didn't check the more obvious answer, which you've already patiently answered before.
Thanks for your help. Now I just need to figure out why some of my questions (like "hi") cause the below. Something for the weekend!
2018-05-12T01:27:14.944034+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args)
2018-05-12T01:27:14.944035+00:00 app[web.1]: File "/app/app.py", line 58, in prediction
2018-05-12T01:27:14.944037+00:00 app[web.1]: response = pred(str(request.json['message']))
2018-05-12T01:27:14.944039+00:00 app[web.1]: File "/app/app.py", line 50, in pred
2018-05-12T01:27:14.944041+00:00 app[web.1]: return model.idsToSentence(ids, wordList)
2018-05-12T01:27:14.944042+00:00 app[web.1]: File "/app/model.py", line 35, in idsToSentence
2018-05-12T01:27:14.944044+00:00 app[web.1]: chosenString = listOfResponses[0]
2018-05-12T01:27:14.945395+00:00 app[web.1]: IndexError: list index out of range
My gut feeling there is that all of the responses/words outputted from the model are empty strings, and thus in line 33 in model.py, listOfResponses will be an empty list and thus, you'll get an index out of range error. You might wanna put some sort of if statement there to check whether the list is empty, and then output something else instead.
That error normally means that your the current model is different from the pre-trained model. Double check to make sure that the shapes of all your variables are the same as the graph you saved, same number of LSTM units, etc.
In the currently live app.py, the code has line 18:
lstmUnits = 48
which conflicts with the code in line 132 of Seq2Seq.py, which is
lstmUnits = 112
Changing both to 112 fixed this issue for me!
Thanks, changed the line in app.py to reflect that.