does it use w_bar when update u and v?
Zzlongjuanfeng opened this issue · comments
in the function _update_u_v(self), does it use w_bar rather than w to update u and v? I mean, should i replace w = getattr(self.module, self.name + '_bar') with w = getattr(self.module, self.name).
That seems reasonable, since the current code appears to be doing a single power iteration update. The original paper indeed updates the most recent `w`. Have you tried running the code with that modification?
Thanks for pointing this out.
… On Jun 14, 2018, at 6:42 AM, zengxianfang ***@***.***> wrote:
in the function _update_u_v(self), does it use w_bar rather than w to update u and v? I mean, should i replace w = getattr(self.module, self.name + '_bar') with w = getattr(self.module, self.name).
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
emmm, not yet. But in the pytorch Implementation, they use the most recent w beacuse they update the weight itself in the line 33.
you're not supposed to update the weight itself, you're supposed to re-normalize the value everytime before using it, without saving the normalization. See Algorithm 1, p15 of the paper