-
pip install -r requirements.txt
to install additional libraries. -
Install deepspeed
See details about config in conf directory
deepspeed --include localhost:0,1,2,3 --no_local_rank distributed_main.py ds_configs=zero3 experiments=sst2 models=gpt-j seed=100
-
Few-shot data file is responsible for Order & Label Balance of demonstrations
- Process order & balance in label sampling stage
- Check sample few-shot data for details
-
Experiment config manages Templates, Verbalizers, Methods, etc.
- Templates include instructions and prompts
- Methods include infernece method(direct or channel)
-
Deepspeed config manages experiment environments including dtype, visible gpus, zero stage, etc.
Run generated_fewshot.py
to randomly sample datasets. We generate train.jsonl
for train set and test.jsonl
for test set. (We just copy the original dataset for test set, only the formatting changes.)
Datasets are saved in json format for each sample.
label
: label of the samplesentence1
: first input of the sample.sentence2
: second input of the sample. For single-sentence tasks,sentence2
is not given.
Parameters
task_name
benchmark_name
: select fromglue
,super_glue
,tweet_eval
,huggingface
huggingface
is for tasks without any specific benchmark (e.g.,trec
,ag_news
)
output_dir
seed
: random seedn_samples
: number of samples (= k)balance
: if given, we sample equal number of samples per class
To run sample scripts : sh sample_scripts/generate_dataset.sh