Question about the model size
fransilvionGenomica opened this issue · comments
Hi,
I am trying to build a Hyena model using hyperparameters from Table A4 (the 4th row). I am using the implementation of a standalone model:
layer2 = HyenaOperator( d_model=1024, l_max=19072, order=36, filter_order=64, num_inner_mlps=4, emb_dim=17, w=14 )
However, when I check the number of parameters, I get ~42M instead of 355M as stated in the paper. Is it because I am using the standalone implementation? But even then how come the difference is so big? Or maybe I am missing something?
`def count_parameters(model):
return sum(p.numel() for p in model.parameters() if p.requires_grad)
count_parameters(layer)`
HyenaOperator is a single layer. “Order 36” refers to the number of
convolutions in the single layer, not the number of layers. To match the
models in the paper, you would want to use order 2, and use 36 of these
layers, along with MLPs and an embedding head (going from vocab size to
embedding dimension).
This is an example of a full language model:
https://github.com/HazyResearch/safari/blob/main/src/models/sequence/long_conv_lm.py
The HyenaOperator functions as the “mixer” in that language model.
…On Fri, Apr 21, 2023 at 11:44 PM fransilvionGenomica < ***@***.***> wrote:
Hi,
I am trying to build a Hyena model using hyperparameters from Table A4
(the 4th row). I am using the implementation of a standalone model:
layer2 = HyenaOperator( d_model=1024, l_max=19072, order=36,
filter_order=64, num_inner_mlps=4, emb_dim=17, w=14 )
However, when I check the number of parameters, I get ~42M instead of 355M
as stated in the paper. Is it because I am using the standalone
implementation? But if that, how come the difference is so big? Or maybe I
am missing something?
`def count_parameters(model):
return sum(p.numel() for p in model.parameters() if p.requires_grad)
count_parameters(layer)`
—
Reply to this email directly, view it on GitHub
<#12>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABDDIITAIMYR24KBFWE7IULXCN43PANCNFSM6AAAAAAXHURWMY>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
Thank you for clarification!