deezer / gravity_graph_autoencoders

Hi, I just noticed while troubleshooting my model that the constructions seem to override the sigmoid activation function with lambda x: x.

See the reference to the Gravity Decoder from inside the GravityGCNModelAE model:

gravity_graph_autoencoders/gravity_gae/model.py

Line 235 in fa09894

self.reconstructions = GravityInspiredDecoder(act = lambda x: x,

It specifies a new activation act = lambda x: x which overrides the sigmoid activation that is written in the paper, and specified as the default act = tf.nn.sigmoid here:

gravity_graph_autoencoders/gravity_gae/layers.py

Line 147 in fa09894

def __init__(self, normalize=False, dropout=0., act=tf.nn.sigmoid, **kwargs):

I guess I'm just wondering whether or not I've misunderstood the implementation somewhere. Is this in fact in error, or is the sigmoid applied elsewhere in the source?

Also, I must add that I'm a big fan of this paper, and also FastGAE! I've been working to incorporate the Gravity decoder together with subgraph sampling for learning on a large, directed dataset. I've really enjoyed reading the papers and working with the source, so thank you for that!

Dear @fleverest,

First of all, thank you very much for your kind message. Your work sounds very interesting, I'd be glad to discuss more about it once you obtain some results!

Regarding your question, I confirm that we need to apply the "identity activation" lambda x: x here. The sigmoid activation from the paper is already incorporated in tf.nn.weighted_cross_entropy_with_logits (more details here) that we subsequently apply in optimizer.py.

Nonetheless, I agree that this act=tf.nn.sigmoid overridden default value in layer.py might be confusing. I even found a related issue on Thomas Kipf's original TF implementation of graph AE/VAE, from which we built our models. I will improve this aspect when time permits.

Best,

Guillaume

Thanks for the instant response! I did see the update earlier, I just haven't had time to respond until now.

I'd be happy to submit a PR with the gravity decoder when I have time to write it up formally.

So this isn't an issue at all, really - in that case you can close this issue, unless you'd like to keep it open as a reminder.

sigmoid activation overridden