JingyunLiang / SwinIR

SwinIR: Image Restoration Using Swin Transformer (official repository)

Home Page:https://arxiv.org/abs/2108.10257

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

about attn_mask and relative_position_bias

YangGangZhiQi opened this issue · comments

Hi, thank you very much for this Great work!
I read your code and paper several times. I can't understand attn_mask and relative_position_bias totally.

In Line 134, why do we just add attn_mask to attn, not multiply? In this way, attn_mask seems like a bias, like relative_popsition_bias in Line 130.
I would appriciate it if someone could help me.

This is because we add the attn_mask (with -100 or 0) before softmax function. A^(x-100) = A^x * A^(-100) = A^x * 0 = 0

@JingyunLiang Thanks for your so quick reply. I get it.
About relative_position_bias, how to understand it? what's the role of relative_position_bias in WindowAttention module? It seems like atten = q @ k + relative_position_bias, what would happen if we do not use relative_position_bias?
I am beginner in SwinIR, Sorry for such questions may take your time.

relative_position_bias can tell you the relative position of two pixels. A pixel should have higher impact on its neighbhouring pixels than distanct pixels.

I get it. Thanks again!

ezoic increase your site revenue