'make_layer_stack' doesn't have a parameter named 'use_universal_transformer'.
danyaljj opened this issue · comments
Getting this error while exporting models:
export PROJECT=...
export ZONE=....
export MODEL_DIR=gs://...
t5_mesh_transformer \
--gcp_project="${PROJECT}" \
--tpu_zone="${ZONE}" \
--model_dir="${MODEL_DIR}" \
--checkpoint_steps=1100500 \
--use_model_api \
--mode="export" \
--export_dir="${MODEL_DIR}/export" \
--tpu="danielk-tpu-europe-west4-a-v3-8-no5"
WARNING:tensorflow:From /home/danielk/ENTER/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:68: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
I0420 19:29:35.249601 140589403665792 mesh_transformer_main.py:141] No write access to model directory. Skipping command logging.
Traceback (most recent call last):
File "/home/danielk/ENTER/bin/t5_mesh_transformer", line 8, in <module>
sys.exit(console_entry_point())
File "/home/danielk/ENTER/lib/python3.7/site-packages/t5/models/mesh_transformer_main.py", line 222, in console_entry_point
app.run(main)
File "/home/danielk/ENTER/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/danielk/ENTER/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "/home/danielk/ENTER/lib/python3.7/site-packages/t5/models/mesh_transformer_main.py", line 200, in main
checkpoint_step=checkpoint_steps)
File "/home/danielk/ENTER/lib/python3.7/site-packages/t5/models/mtf_model.py", line 360, in export
gin.parse_config_file(_operative_config_path(self._model_dir))
File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/config.py", line 1702, in parse_config_file
parse_config(f, skip_unknown=skip_unknown)
File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/config.py", line 1611, in parse_config
bind_parameter((scope, selector, arg_name), value)
File "/home/danielk/ENTER/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/utils.py", line 68, in try_with_location
augment_exception_message_and_reraise(exception, _format_location(location))
File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise
six.raise_from(proxy.with_traceback(exception.__traceback__), None)
File "<string>", line 3, in raise_from
File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/utils.py", line 66, in try_with_location
yield
File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/config.py", line 1611, in parse_config
bind_parameter((scope, selector, arg_name), value)
File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/config.py", line 648, in bind_parameter
pbk = ParsedBindingKey(binding_key)
File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/config.py", line 517, in __new__
raise ValueError(err_str.format(selector, arg_name))
ValueError: Configurable 'make_layer_stack' doesn't have a parameter named 'use_universal_transformer'.
In file "gs://danielk-files/t5-models/union_mixture/3B/operative_config.gin", line 83
decoder/make_layer_stack.use_universal_transformer = False
Versions for completeness:
t5 0.5.0
tensorboard 1.15.0
tensorflow 1.15.2
tensorflow-datasets 3.0.0
tensorflow-estimator 1.15.1
tensorflow-metadata 0.21.2
tensorflow-text 1.15.0
torch 1.4.0
I used the same command to export models a couple of weeks ago. So I suspect a recent change could be causing this error.
Update: I think using a slightly older version of tensorflow-mesh (https://pypi.org/project/mesh-tensorflow/0.1.12/) the error goes away.
Hi Daniel, yes, unfortunately there was a change introduced in the Mesh TF Transformer that will break any operative config saved during a short period of time. It should be fine to just remove that line from the offending operative config and use the latest MTF.