horovod / horovod

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

Home Page:http://horovod.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Test test.integration.test_spark.SparkTests.test_dbfs_local_store broken for tensorflow>=2.13

EnricoMi opened this issue · comments

Test fails for Tensorflow 2.13.0 and above:

                reconstructed_model_local = deserialize_keras_model(serialized_model_local)
                if version.parse(tensorflow.__version__) >= version.parse("2.3.0"):
>                   assert reconstructed_model_local.get_config() == model.get_config()
E                   AssertionError: assert {'layers': [{... 'sequential'} == {'layers': [{... 'sequential'}
E                     Omitting 1 identical items, use -vv to show
E                     Differing items:
E                     {'layers': [{'class_name': 'InputLayer', 'config': {'batch_input_shape': (None, 32), 'dtype': 'float32', 'name': 'inpu...'Zeros', 'config': {}, 'module': 'keras.initializers', 'registered_name': None}, ...}, 'module': 'keras.layers', ...}]} != {'layers': [{'class_name': 'InputLayer', 'config': {'batch_input_shape': (None, 32), 'dtype': 'float32', 'name': 'inpu...s_initializer': {'class_name': 'Zeros', 'config': {}, 'module': 'keras.initializers', 'registered_name': None}, ...}}]}
E                     Full diff:
E                       {
E                        'layers': [{'clas...
E                     
E                     ...Full output truncated (42 lines hidden), use '-vv' to show
``

@liangz1 @irasit @WeichenXu123 I am going to disable this test for Tensorflow 2.13.0 and above, which means we are loosing test coverage for your logic.