configs for Hyena Wikitext103 experiments

Question

configs for Hyena Wikitext103 experiments

xiaobo-guo opened this issue a year ago · comments

Your work is excellent! I am trying to follow your work and facing some problems. I wonder if you may share the config for the wiki103 dataset of the Hyena. I try to conduct experiments with 125-slim but the test perplexity is higher than the reported result (about 21 with hyena). And I am wondering whether the removal of flash-atten will influence the result or not.

Michael Poli · Answer 1 · Sat Jun 24 2023 02:34:50 GMT+0800 (China Standard Time)

Can you share the config? Wikitext is quite sensitive to a few hyperparameters. Flash attention will not affect the result for Hyena.

Xiaobo Guo · Answer 2 · Sat Jun 24 2023 02:59:40 GMT+0800 (China Standard Time)

Thanks for your response.

I attach the config file
config.txt

Michael Poli · Answer 3 · Sat Jun 24 2023 03:01:23 GMT+0800 (China Standard Time)

You should set dropouts to 0.2 as a first step. After you get to sub 19 ppl you will be in tuning range.

Xiaobo Guo · Answer 4 · Sat Jun 24 2023 03:05:21 GMT+0800 (China Standard Time)

Thank you, shall I also set the order to be 3 in the Heyna layer?

Songlin Yang · Answer 5 · Wed Jun 28 2023 11:42:00 GMT+0800 (China Standard Time)

Can you share the config? Wikitext is quite sensitive to a few hyperparameters. Flash attention will not affect the result for Hyena.

Could you please put the configs you used in configs/experiment/wt103? That would be super helpful!

Songlin Yang · Answer 6 · Thu Jun 29 2023 21:48:10 GMT+0800 (China Standard Time)

Thank you, shall I also set the order to be 3 in the Heyna layer?

Did you reproduce the 19 ppl result using dropout=0.2? I still get 22

Xiaobo Guo · Answer 7 · Thu Jun 29 2023 23:31:08 GMT+0800 (China Standard Time)

Thank you, shall I also set the order to be 3 in the Heyna layer?

Did you reproduce the 19 ppl result using dropout=0.2? I still get 22

I set the dropout to 0.2 and the order to 3 and get about 20 but still can not get the reported result

Michael Poli · Answer 8 · Thu Jun 29 2023 23:41:09 GMT+0800 (China Standard Time)

You can look at this config for an independent reproduction that gets to sub 19. Let me know if after this you still have issues with the loss being too high, and I'll rerun experiments in the new codebase.

Shida Wang · Answer 9 · Fri Jun 30 2023 14:42:37 GMT+0800 (China Standard Time)

Thank you, shall I also set the order to be 3 in the Heyna layer?

Did you reproduce the 19 ppl result using dropout=0.2? I still get 22

I set the dropout to 0.2 and the order to 3 and get about 20 but still can not get the reported result

Question:
Did you change the attain_layer_indx? It seems in your attached config, there is attention layer at layer 1 and 8 (which is inherited from the base.yaml)

Shida Wang · Answer 10 · Fri Jul 07 2023 10:29:54 GMT+0800 (China Standard Time)

You can look at this config for an independent reproduction that gets to sub 19. Let me know if after this you still have issues with the loss being too high, and I'll rerun experiments in the new codebase.

Thanks for the helpful reference. However, I checked that repo and the released [log] from S5 (https://wandb.ai/jimmysmith1919/S5_ICL/reports/Hyena-red-and-Hyena-S5-blue-on-WikiText-103--Vmlldzo0MTkwODEx?accessToken=pk0zw5w75uo1s4zkn3kh7koum902t4q2yzbm28xk0olzzgxuskoq0g1iyauixlob) which shows that Hyena with test perplexity 19.094.

It would be very helpful if you can share the detailed configuration of Hyena on wikitext-103.