Giters

sdc17 / UPop

[ICML 2023] UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers.

Home Page:https://dachuanshi.com/UPop-Project/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Problem with CLIP implementation

jiseshen opened this issue · comments

commented

There's a bug concerning CLIP implementation, since nn.MultiheadAttention does not have a parameter called "search", which is customized in your implementation of other modules to give the model a learnable pruning parameter. However, the command runs without error. I will appreciate it if you would help me with that.

commented

Hi, @jiseshen this is not a bug. When running the code, clip/mock.py dynamically replaces some functions of nn.MultiheadAttention with our own implementations and inserts new parameters UPop needs. For example, this line

UPop/clip/mock.py

Line 248 in 4ddd4bb

kdim=None, vdim=None, batch_first=False, device=None, dtype=None, search=False) -> None:

inserts the "search" into nn.MultiheadAttention. Some related explanations are also mentioned in #20.