Revamp experiment organization and migrate code when necessary

[Stella Biderman]

We’ve been pushing to get experiments for several papers done, but now that the Pythia and Memorization papers are both done we should take some time to refactor and reorganize this repo. My vision is that it will serve as the central organization point for anyone interested in replicating or extending the work we do with Pythia.

I’m going to spend today thinking about how to best arrange things, and update this issue on Monday with my thoughts.

Known To-Do Items

• Identify the correct runs on WandB, clearly label them, and organize a report showing them + evals.

• Some of the visualization code for the Memorization paper is still only on @StellaAthena’s laptop. She needs to clean it up and upload it.

• While the experimental data is better organized than it was, it’s still not very well organized. We can do better.