tensorflow.python.framework.errors_impl.AlreadyExistsError: Another metric with the same name already exists.
jarmstrongcorus opened this issue · comments
I'm encountering an issue in Dataflow when I try to run a simple pipeline. I think this is perhaps related to a Tensorflow 2.7 TFX compatibility issue, but I don't have insight into what Dataflow is using, and I've encountered issues with trying to pass it a requirements.txt file.
The Dataflow configuration is running
tfx_executor:
tfx-components-example_gen-csv_example_gen-executor-executor
tfx_py_version:
3-7
tfx_version:
1-3-0
Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/apache_beam/internal/pickler.py", line 285, in loads return dill.loads(s) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 275, in loads return load(file, ignore, **kwds) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 270, in load return Unpickler(file, ignore=ignore, **kwds).load() File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 472, in load obj = StockUnpickler.load(self) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 462, in find_class return StockUnpickler.find_class(self, module, name) File "/usr/local/lib/python3.7/site-packages/tfx/components/example_gen/csv_example_gen/executor.py", line 22, in from tfx.components.example_gen.base_example_gen_executor import BaseExampleGenExecutor File "/usr/local/lib/python3.7/site-packages/tfx/components/example_gen/base_example_gen_executor.py", line 31, in from tfx.types import standard_component_specs File "/usr/local/lib/python3.7/site-packages/tfx/types/standard_component_specs.py", line 16, in import tensorflow_model_analysis as tfma File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/init.py", line 33, in from tensorflow_model_analysis.api import tfma_unit as test File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/api/tfma_unit.py", line 71, in from tensorflow_model_analysis.api import model_eval_lib File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/api/model_eval_lib.py", line 36, in from tensorflow_model_analysis.evaluators import evaluator File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/evaluators/init.py", line 17, in from tensorflow_model_analysis.evaluators.analysis_table_evaluator import AnalysisTableEvaluator File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/evaluators/analysis_table_evaluator.py", line 27, in from tensorflow_model_analysis.evaluators import evaluator File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/evaluators/evaluator.py", line 25, in from tensorflow_model_analysis.extractors import extractor File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/init.py", line 18, in from tensorflow_model_analysis.extractors.example_weights_extractor import ExampleWeightsExtractor File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/example_weights_extractor.py", line 27, in from tensorflow_model_analysis.extractors import extractor File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/extractor.py", line 26, in from tensorflow_model_analysis.utils import util File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/utils/init.py", line 21, in from tensorflow_model_analysis.utils.model_util import CombineFnWithModels File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/utils/model_util.py", line 31, in from tensorflow_model_analysis.eval_saved_model import load File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py", line 25, in from tensorflow_model_analysis.eval_metrics_graph import eval_metrics_graph File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/eval_metrics_graph/eval_metrics_graph.py", line 44, in from tensorflow_model_analysis.eval_saved_model import util File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/util.py", line 492, in checkpoint_path: Optional[Text] = None) -> Optional[bytes]: File "/usr/local/lib/python3.7/site-packages/tensorflow/python/util/lazy_loader.py", line 62, in getattr module = self._load() File "/usr/local/lib/python3.7/site-packages/tensorflow/python/util/lazy_loader.py", line 45, in _load module = importlib.import_module(self.name) File "/usr/local/lib/python3.7/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/api/init.py", line 10, in from tensorflow_estimator.python.estimator.api._v1 import estimator File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/api/_v1/estimator/init.py", line 10, in from tensorflow_estimator.python.estimator.api._v1.estimator import experimental File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/api/_v1/estimator/experimental/init.py", line 10, in from tensorflow_estimator.python.estimator.canned.dnn import dnn_logit_fn_builder File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/dnn.py", line 27, in from tensorflow_estimator.python.estimator import estimator File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 70, in @doc_controls.inheritable_header("""\ AttributeError: module 'tensorflow.tools.docs.doc_controls' has no attribute 'inheritable_header' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 651, in do_work work_executor.execute() File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 179, in execute op.start() File "apache_beam/runners/worker/operations.py", line 704, in apache_beam.runners.worker.operations.DoOperation.start File "apache_beam/runners/worker/operations.py", line 706, in apache_beam.runners.worker.operations.DoOperation.start File "apache_beam/runners/worker/operations.py", line 707, in apache_beam.runners.worker.operations.DoOperation.start File "apache_beam/runners/worker/operations.py", line 305, in apache_beam.runners.worker.operations.Operation.start File "apache_beam/runners/worker/operations.py", line 311, in apache_beam.runners.worker.operations.Operation.start File "apache_beam/runners/worker/operations.py", line 653, in apache_beam.runners.worker.operations.DoOperation.setup File "apache_beam/runners/worker/operations.py", line 654, in apache_beam.runners.worker.operations.DoOperation.setup File "apache_beam/runners/worker/operations.py", line 286, in apache_beam.runners.worker.operations.Operation.setup File "apache_beam/runners/worker/operations.py", line 300, in apache_beam.runners.worker.operations.Operation.setup File "apache_beam/runners/worker/operations.py", line 793, in apache_beam.runners.worker.operations.DoOperation._get_runtime_performance_hints File "/usr/local/lib/python3.7/site-packages/apache_beam/internal/pickler.py", line 289, in loads return dill.loads(s) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 275, in loads return load(file, ignore, **kwds) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 270, in load return Unpickler(file, ignore=ignore, **kwds).load() File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 472, in load obj = StockUnpickler.load(self) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 462, in find_class return StockUnpickler.find_class(self, module, name) File "/usr/local/lib/python3.7/site-packages/tfx/components/example_gen/csv_example_gen/executor.py", line 22, in from tfx.components.example_gen.base_example_gen_executor import BaseExampleGenExecutor File "/usr/local/lib/python3.7/site-packages/tfx/components/example_gen/base_example_gen_executor.py", line 31, in from tfx.types import standard_component_specs File "/usr/local/lib/python3.7/site-packages/tfx/types/standard_component_specs.py", line 16, in import tensorflow_model_analysis as tfma File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/init.py", line 33, in from tensorflow_model_analysis.api import tfma_unit as test File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/api/tfma_unit.py", line 71, in from tensorflow_model_analysis.api import model_eval_lib File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/api/model_eval_lib.py", line 36, in from tensorflow_model_analysis.evaluators import evaluator File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/evaluators/init.py", line 17, in from tensorflow_model_analysis.evaluators.analysis_table_evaluator import AnalysisTableEvaluator File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/evaluators/analysis_table_evaluator.py", line 27, in from tensorflow_model_analysis.evaluators import evaluator File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/evaluators/evaluator.py", line 25, in from tensorflow_model_analysis.extractors import extractor File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/init.py", line 18, in from tensorflow_model_analysis.extractors.example_weights_extractor import ExampleWeightsExtractor File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/example_weights_extractor.py", line 27, in from tensorflow_model_analysis.extractors import extractor File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/extractor.py", line 26, in from tensorflow_model_analysis.utils import util File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/utils/init.py", line 21, in from tensorflow_model_analysis.utils.model_util import CombineFnWithModels File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/utils/model_util.py", line 31, in from tensorflow_model_analysis.eval_saved_model import load File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py", line 25, in from tensorflow_model_analysis.eval_metrics_graph import eval_metrics_graph File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/eval_metrics_graph/eval_metrics_graph.py", line 44, in from tensorflow_model_analysis.eval_saved_model import util File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/util.py", line 492, in checkpoint_path: Optional[Text] = None) -> Optional[bytes]: File "/usr/local/lib/python3.7/site-packages/tensorflow/python/util/lazy_loader.py", line 62, in getattr module = self._load() File "/usr/local/lib/python3.7/site-packages/tensorflow/python/util/lazy_loader.py", line 45, in _load module = importlib.import_module(self.name) File "/usr/local/lib/python3.7/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/api/init.py", line 10, in from tensorflow_estimator.python.estimator.api._v1 import estimator File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/api/_v1/estimator/init.py", line 10, in from tensorflow_estimator.python.estimator.api._v1.estimator import experimental File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/api/_v1/estimator/experimental/init.py", line 10, in from tensorflow_estimator.python.estimator.canned.dnn import dnn_logit_fn_builder File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/dnn.py", line 27, in from tensorflow_estimator.python.estimator import estimator File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 62, in 'estimator api usage', 'method') File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/monitoring.py", line 361, in init len(labels), name, description, *labels) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/monitoring.py", line 135, in init self._metric = self._metric_methods[self._label_length].create(*args) tensorflow.python.framework.errors_impl.AlreadyExistsError: Another metric with the same name already exists.
We see that you are using old version of TFX , We recommend that you upgrade to 1.4.0 and let us know if the issue still persists in newer versions and also refer compatible versions doc. Thanks!
