EleutherAI / lm-evaluation-harness

What

Hugging Face Datasets support Slice Splits

datase_10pc = datasets.load_dataset("mydataset", split="test[:10%]")

Therefore I assumed that when creating a new task, I could express the dataset split as:

task: mytask
dataset_path: user/mydataset
dataset_name: null
training_split: null
validation_split: null
test_split: 'test[:50%]'
doc_to_text: abc
doc_to_target: def
metric_list:
  - metric: ...

However, that fails in:

lm-evaluation-harness/lm_eval/api/task.py

Line 901 in 30c060d

return self.dataset[self.config.training_split]

with KeyError: 'test[:50%]'

Support loading slices of a split from a dataset

What