Giters

EleutherAI / pythia

The hub for EleutherAI's work on interpretability and learning dynamics

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Complete outstanding evals

haileyschoelkopf opened this issue · comments

commented

Hailey:
0-shot:

  • EleutherAI/pythia-v1.1-70m
  • EleutherAI/pythia-v1.1-70m-deduped
  • EleutherAI/pythia-v1.1-160m
  • EleutherAI/pythia-v1.1-160m-deduped
  • EleutherAI/pythia-v1.1-410m
  • EleutherAI/pythia-v1.1-410m-deduped
  • EleutherAI/pythia-v1.1-1b
  • EleutherAI/pythia-v1.1-1b-deduped
  • EleutherAI/pythia-v1.1-1.4b
  • EleutherAI/pythia-v1.1-1.4b-deduped
  • EleutherAI/pythia-v1.1-2.8b
  • EleutherAI/pythia-v1.1-2.8b-deduped
  • EleutherAI/pythia-v1.1-6.9b
  • EleutherAI/pythia-v1.1-6.9b-deduped
  • EleutherAI/pythia-v1.1-12b (missing 43000, 123000, 133000, 143000)
  • EleutherAI/pythia-v1.1-12b-deduped (missing 33000, 43000, 53000, 63000, 73000, 83000, 93000)
  • interventions

Sai:
steps equivalent to [3000,13000,....,123000,133000,143000]

  • EleutherAI/pythia-v1.1-70m-0.25MtokBS (model total steps: 1144000)
  • EleutherAI/pythia-v1.1-160m-0.5MtokBS (model total steps: 572000)
  • EleutherAI/pythia-v1.1-410m-0.5MtokBS (model total steps: 572000)
  • EleutherAI/pythia-v1.1-1b-0.5MtokBS (model total steps: 572000)
  • EleutherAI/pythia-v1.1-1.4b-1MtokBS (model total steps: 286000)

Aviya:
5-shot, steps [3000,13000,....,123000,133000,143000]:

  • EleutherAI/pythia-v1.1-70m
  • EleutherAI/pythia-v1.1-70m-deduped
  • EleutherAI/pythia-v1.1-160m
  • EleutherAI/pythia-v1.1-160m-deduped
  • EleutherAI/pythia-v1.1-410m
  • EleutherAI/pythia-v1.1-410m-deduped
  • EleutherAI/pythia-v1.1-1b
  • EleutherAI/pythia-v1.1-1b-deduped
  • EleutherAI/pythia-v1.1-1.4b
  • EleutherAI/pythia-v1.1-1.4b-deduped
  • EleutherAI/pythia-v1.1-2.8b
  • EleutherAI/pythia-v1.1-2.8b-deduped
  • EleutherAI/pythia-v1.1-6.9b
  • EleutherAI/pythia-v1.1-6.9b-deduped
  • EleutherAI/pythia-v1.1-12b
  • EleutherAI/pythia-v1.1-12b-deduped

Herbie?

  • Winobias on interventions + baseline

Task list:

  • hendrycksTest*
  • piqa
  • sciq
  • lambada_openai
  • winogrande
  • wsc
  • arc_challenge
  • arc_easy
  • logiqa
  • crows_pairs_*

Models missing still:

  • EleutherAI/intervention-pythia-v1.1-1.4b (MISSING)
  • EleutherAI/intervention-pythia-v1.1-1.4b-long (MISSING)
  • EleutherAI/pythia-v1.1-12b (steps after 123000 are missing)
  • EleutherAI/pythia-v1.1-1b (MISSING)
  • EleutherAI/pythia-v1.1-410m-0.5MtokBS
  • EleutherAI/pythia-v1.1-1b-0.5MtokBS
  • EleutherAI/pythia-v1.1-1.4b-1MtokBS