Reproduce Flickr30k Evaluation results - DataSet problem
JoanFM opened this issue · comments
Hello again @dandelin ,
I was trying to reproduce the steps from https://github.com/dandelin/ViLT/blob/master/EVAL.md to get the results from Flickr30k T2IR.
First I did what is suggested in https://github.com/dandelin/ViLT/blob/master/DATA.md.
So I have in a folder /content/flickr30k this structure:
/content/flickr30k
├── flickr30k_images
│ ├── ....jpg
| ├── ....jpg
├── karpathy
├── dataset_flickr30k.json
Then I do the transformation:
from vilt.utils.write_f30k_karpathy import make_arrow
make_arrow( '/content/flickr30k', '/content/arrow')But when I run:
python run.py with data_root='/content/arrow' num_gpus=1 num_nodes=1 per_gpu_batchsize=4 task_finetune_irtr_f30k_randaug test_only=True load_path="/content/TFM_Sparse_Embeddings/vilt_irtr_f30k.ckpt"
I get the error:
ERROR - ViLT - Failed after 0:00:06!
Traceback (most recent calls WITHOUT Sacred internals):
File "run.py", line 73, in main
trainer.test(model, datamodule=dm)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 755, in test
results = self.__test_given_model(model, test_dataloaders)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 820, in __test_given_model
results = self.fit(model)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 473, in fit
results = self.accelerator_backend.train()
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 152, in train
results = self.ddp_train(process_idx=self.task_idx, model=model)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 305, in ddp_train
results = self.train_or_test()
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 67, in train_or_test
results = self.trainer.run_test()
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 662, in run_test
eval_loop_results, _ = self.run_evaluation(test_mode=True)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 566, in run_evaluation
dataloaders, max_batches = self.evaluation_loop.get_evaluation_dataloaders(max_batches)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 56, in get_evaluation_dataloaders
self.trainer.reset_test_dataloader(model)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/data_loading.py", line 299, in reset_test_dataloader
self._reset_eval_dataloader(model, 'test')
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/data_loading.py", line 249, in _reset_eval_dataloader
num_batches = len(dataloader) if has_len(dataloader) else float('inf')
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/data.py", line 33, in has_len
raise ValueError('`Dataloader` returned 0 length.'
ValueError: `Dataloader` returned 0 length. Please make sure that your Dataloader at least returns 1 batch
Hi @JoanFM
Could you confirm that files named f30k_caption_karpathy_train.arrow, f30k_caption_karpathy_val.arrow, f30k_caption_karpathy_test.arrow exist under the /content/arrow directory?