tf-encrypted / tf-encrypted

A Framework for Encrypted Machine Learning in TensorFlow

Home Page:https://tf-encrypted.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

AUC problem in logistic regression examples

wuxuedaifu opened this issue · comments

I tried to add roc_auc function in common.py file, and uses different datasetes to train.
No matter how many times I tried, the AUC result is always around 0.5.

However after revising the common.py file, make the calculation iteration in tensorflow mode (replacing "TFE" API with "TF" API) module, the auc value goes back to 0.8.

def compute_auc(self, sess, x, y, data_owner):
    # FIXME: buggy
    def print_auc(y_hat, y_in):
        y_hat_list = list(y_hat.eval())
        y_in_list = list(y_in.eval())
        s = '\n'.join([str(x) for x in y_hat_list])
        fin = open(r'y_hat.txt', 'w')
        fin.write(s)
        fin.close()

        s = '\n'.join([str(x) for x in y_in_list])
        fin = open(r'y_in.txt', 'w')
        fin.write(s)
        fin.close()

        print(y_hat.eval())
        print("======================================")
        print(y_in.eval())

        with tf.name_scope("print_auc"):
            auc = roc_auc_score(y_in.eval(), y_hat.eval())

            #auc, update_auc_op = tf.metrics.auc(y_in, y_hat)
            op = tf.print("AUC on {}:".format(data_owner.player_name),
                          auc, type(y_in), type(y_hat), summarize=6)
            return op

    with tf.name_scope("auc"):
        y_p = self.forward(x)
        print_auc_op = tfe.define_output(data_owner.player_name,
                                         [y_p, y],
                                         print_auc)
    print(f'local variables: {tf.get_collection(tf.GraphKeys.LOCAL_VARIABLES)}')
    sess.run(tf.initialize_local_variables())
    sess.run(print_auc_op)

    # compute auc with revealed prediction and label
    y_pred = y_p
    print(type(y_pred), y_pred.shape, f'y pred: {y_pred}')
    y_true = y
    print(type(y_true), y_true.shape, f'y true: {y_true}')
    # acc = np.isclose(np.around(y_pred), y_true)
    # print(f'Acc {np.mean(acc)}')
    # print(f'AUC {roc_auc_score(y_true, y_pred)}')
    return y_pred, y_true

I doubt the there is some problem in the TFE calculation process.
Need help!!

I found out the problem:

using tfe.define_output(data_owner.player_name,
[y_p, y],
print_auc)

to evaluate the model is not correct.
The trained parameters is saved in LogisticRegressionModel.weights (tuple), so take the weights out using sess.run(model.weights.reval()), and then use this weights to calcualte the accuracy and auc value

thanks for the report, looks like we can close this up since you've already resolved it