Ablate on initialization
mitchellnw opened this issue · comments
interested in:
A) changing layer_id + 1 to args.num_layers.
B) removing the line std = std / math.sqrt(2 * (layer_id + 1))
related: #225
We tested #225 at 1B and it seems to hurt downstream evals significantly, unfortunately.