Giters

ggerganov / llama.cpp

LLM inference in C/C++

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

GGML_ASSERT(n_embd_gqa == n_embd_k_gqa) fails in models where key vector dimension is different from value vector dimension

fairydreaming opened this issue · comments

commented

During my work on DeepSeek-V2 I noticed a failing assert:

llama.cpp/llama.cpp

Line 4923 in 9afdffe

GGML_ASSERT(n_embd_gqa == n_embd_k_gqa);

Since n_embd_gqa is set to n_embd_v_gqa this assert only works for models where n_embd_k_gqa == n_embd_v_gqa, that is when n_embd_head_k == n_embd_head_v - and it fails when n_embd_head_k != n_embd_head_v. Is this intentional behavior?