global layer normalization

[Fei Jiang]

Hi, Efthymios
Thanks for sharing the code!
Is this code

sudo_rm_rf/sudo_rm_rf/dnn/models/sudormrf.py

Line 116 in 5bf8c48

 self.norm = nn.GroupNorm(1, nOut, eps=1e-08) 

the "global layer normalization" mentioned in the paper?

[Efthymios Tzinis]

No this is layernorm. Check the improved version where there is also the global norm layer https://github.com/etzinis/sudo_rm_rf/blob/5bf8c48d091c6adc74f834098692dc0386d961f0/sudo_rm_rf/dnn/models/improved_sudormrf.py

…

On Tue, Jul 13, 2021 at 7:38 AM Fei Jiang ***@***.***> wrote: Hi, Efthymios Thanks for sharing the code! Is this code https://github.com/etzinis/sudo_rm_rf/blob/5bf8c48d091c6adc74f834098692dc0386d961f0/sudo_rm_rf/dnn/models/sudormrf.py#L116 the "global layer normalization" mentioned in the paper? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFOOZCVBJGOOXLKO6UHED63TXQXVVANCNFSM5AJAUDQQ> .

--

__________________________________________________________________ Efthymios Tzinis University of Illinois at Urbana Champaign (UIUC) PhD Candidate in Computer Science Siebel Office: 3332 | 201 N. Goodwin Ave. Urbana, IL, 61801, USA

[Fei Jiang]

I think this implementation makes no difference with nn.GroupNorm(1, nOut, eps=1e-08). Here is the test code:
import torch
import torch.nn as nn

x = torch.rand(2, 4, 3)
ln = nn.GroupNorm(1, 4, eps=1e-8)
print(ln(x))
print((x-x.mean(dim=[1, 2], keepdims=True))/(x.var(dim=[1, 2], keepdims=True, unbiased=False)+1e-8).sqrt()) # This is equivalent to your implementation except for the affine operation