why change n_copies from 1 to 2?
Reeleon opened this issue · comments
【Related Code】
bigcode_eval/utils.py
class TokenizedDataset(IterableDataset):
def __iter__(self):
...
if self.n_copies == 1 and self.n_tasks % self.num_devices != 0:
self.n_copies = 2
warnings.warn(
"n_copies (n_samples/batch_size) was changed from 1 to 2 because n_tasks isn't proportional to num devices"
)
...
【My Setting】
n_tasks = 164 # humaneval
num_devices = 5
n_samples = 1
batch_size = 1
【Results】
I get the UserWarning: n_copies (n_samples/batch_size) was changed from 1 to 2 because n_tasks isn't proportional to num devices
.
And the harness generated 2 samples
for 163 tasks
,and 4 samples
for 1 tasks
before removed extra predictions to only keep nsamples=1
。
Why not keep n_copies=1
,and just generate 1 samples
for 163 tasks
and 2 samples
for 1 tasks
?It make n_tasks = 1 * 163 + 2 * 1 = 165
, which also ensure n_tasks % num_devices == 0
.