Model Name
conceptofmind opened this issue · comments
@lucidrains What would you like the first model to be named?
One restart. 160B tokens.
Did some tests with qk_norm vs no qk_norm as well. When using AdamW decided to go with qk_norm=False. I will explore this with Lion after.
PaLM 1B