Is it possible to evaluate the BaseModel with other keras metrics aside from binary and categorical accuracy?
fbarusso opened this issue · comments
I would like to evaluate a private model (to be specific the true/false negatives/positives metrics), but the evaluate function from the BaseModel seems to support only binary and categorical accuracy. I'm using a logistic regression model for binary classification.
@zicofish Thanks for your attention! I tried to use the benchmark/private_training.py by running:
tf-encrypted/examples/benchmark/training$ ./run-remote.sh logistic_regression LogisticArtificial
But on line 133, I used
result = model_2.predict(x=test_data_iter)
instead of
result = model_2.evaluate(x=test_data_iter, metrics=metrics, steps=test_dataset.iterations)
and it resulted in the following error:
Traceback (most recent call last):
File "./private_training.py", line 133, in <module>
result = model_2.predict(x=test_data_iter)
File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/base_model.py", line 146, in predict
y_pred = self.predict_function(input_x)
File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/protocol/protocol.py", line 188, in wrap_function
result = graph_function(args, kwargs)
File "/home/febarusso/miniconda3/envs/tfe/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/tmp/__autograph_generated_filej69ox2qn.py", line 12, in tf__graph_function
result = ag__.converted_call(ag__.ld(func), tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)
File "/tmp/__autograph_generated_filezpzu8lwk.py", line 12, in tf__predict_step
y_pred = ag__.converted_call(ag__.ld(self).call, (ag__.ld(input_x),), None, fscope)
File "/tmp/__autograph_generated_filegmnf96cf.py", line 51, in tf__call
ag__.for_stmt(ag__.ld(self).layers, None, loop_body, get_state_2, set_state_2, ('outputs', 'inputs'), {'iterate_names': 'layer'})
File "/tmp/__autograph_generated_filegmnf96cf.py", line 48, in loop_body
outputs = ag__.converted_call(ag__.ld(layer), (ag__.ld(inputs),), None, fscope)
File "/tmp/__autograph_generated_file_c7f9xgv.py", line 28, in tf____call__
outputs = ag__.converted_call(ag__.ld(self).call, ((ag__.ld(inputs),) + tuple(ag__.ld(args))), dict(**ag__.ld(kargs)), fscope)
File "/tmp/__autograph_generated_file43d06n4z.py", line 11, in tf__call
input_shape = ag__.converted_call(ag__.ld(inputs).shape.as_list, (), None, fscope)
AttributeError: in user code:
File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/protocol/protocol.py", line 178, in graph_function *
result = func(*args, **kwargs)
File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/base_model.py", line 129, in predict_step *
y_pred = self.call(input_x)
File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/sequential.py", line 109, in call *
outputs = layer(inputs)
File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/engine/base_layer.py", line 110, in __call__ *
outputs = self.call(inputs, *args, **kargs)
File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/layers/flatten.py", line 33, in call *
input_shape = inputs.shape.as_list()
AttributeError: 'generator' object has no attribute 'shape'
@zicofish Thanks for your attention! I tried to use the benchmark/private_training.py by running:
tf-encrypted/examples/benchmark/training$ ./run-remote.sh logistic_regression LogisticArtificialBut on line 133, I used
result = model_2.predict(x=test_data_iter)instead of
result = model_2.evaluate(x=test_data_iter, metrics=metrics, steps=test_dataset.iterations)and it resulted in the following error:
Traceback (most recent call last): File "./private_training.py", line 133, in <module> result = model_2.predict(x=test_data_iter) File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/base_model.py", line 146, in predict y_pred = self.predict_function(input_x) File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/protocol/protocol.py", line 188, in wrap_function result = graph_function(args, kwargs) File "/home/febarusso/miniconda3/envs/tfe/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler raise e.with_traceback(filtered_tb) from None File "/tmp/__autograph_generated_filej69ox2qn.py", line 12, in tf__graph_function result = ag__.converted_call(ag__.ld(func), tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope) File "/tmp/__autograph_generated_filezpzu8lwk.py", line 12, in tf__predict_step y_pred = ag__.converted_call(ag__.ld(self).call, (ag__.ld(input_x),), None, fscope) File "/tmp/__autograph_generated_filegmnf96cf.py", line 51, in tf__call ag__.for_stmt(ag__.ld(self).layers, None, loop_body, get_state_2, set_state_2, ('outputs', 'inputs'), {'iterate_names': 'layer'}) File "/tmp/__autograph_generated_filegmnf96cf.py", line 48, in loop_body outputs = ag__.converted_call(ag__.ld(layer), (ag__.ld(inputs),), None, fscope) File "/tmp/__autograph_generated_file_c7f9xgv.py", line 28, in tf____call__ outputs = ag__.converted_call(ag__.ld(self).call, ((ag__.ld(inputs),) + tuple(ag__.ld(args))), dict(**ag__.ld(kargs)), fscope) File "/tmp/__autograph_generated_file43d06n4z.py", line 11, in tf__call input_shape = ag__.converted_call(ag__.ld(inputs).shape.as_list, (), None, fscope) AttributeError: in user code: File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/protocol/protocol.py", line 178, in graph_function * result = func(*args, **kwargs) File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/base_model.py", line 129, in predict_step * y_pred = self.call(input_x) File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/models/sequential.py", line 109, in call * outputs = layer(inputs) File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/engine/base_layer.py", line 110, in __call__ * outputs = self.call(inputs, *args, **kargs) File "/home/febarusso/TFE/tf-encrypted/tf_encrypted/keras/layers/flatten.py", line 33, in call * input_shape = inputs.shape.as_list() AttributeError: 'generator' object has no attribute 'shape'
@fbarusso you should not use the same test_data_iter for predict, because test_data_iter contains both x an y, which is good for evaluate, but not predict. You might need to figure out some way to extract only x. For example, in the predict function:
changing for input_x in data_iter: to for (input_x, input_y) in data_iter: will make it work. But there should be a better way to achieve what you want.
