Cannot reproduce the results for 1-shot 5-way Mini-ImageNet
xwjabc opened this issue · comments
I used the command below to perform the experiment (environment: Python 3.6 / TensorFlow 1.10):
# 1-shot 5-way Mini-ImageNet.
python -u run_miniimagenet.py --shots 1 --inner-batch 10 --inner-iters 8 --meta-step 1 --meta-batch 5 --meta-iters 100000 --eval-batch 15 --eval-iters 50 --learning-rate 0.001 --meta-step-final 0 --train-shots 5 --checkpoint ckpt_m15t
The last several lines of output:
batch 99950: train=0.200000 test=0.200000
batch 99960: train=0.000000 test=0.000000
batch 99970: train=0.200000 test=0.000000
batch 99980: train=0.200000 test=0.200000
batch 99990: train=0.200000 test=0.400000
Evaluating...
Train accuracy: 0.28016
Validation accuracy: 0.26512
Test accuracy: 0.24262
TensorBoard output:
However, in the paper, the test accuracy is around 47%.
It seems the dataset is indeed incomplete (see the output below). However, it does not lose too many images (38392/38400, 9593/9600, 11996/12000). Thus, I wonder if there are some other reasons behind the performance.
/m/c/w/c/F/R/supervised-reptile/data/miniimagenet
find ./train -name '*.JPEG' | wc -l
38392
/m/c/w/c/F/R/supervised-reptile/data/miniimagenet
find ./val -name '*.JPEG' | wc -l
9593
/m/c/w/c/F/R/supervised-reptile/data/miniimagenet
find ./test -name '*.JPEG' | wc -l
11996
By the way, could you send me a copy of the data? Thank you very much!
See if 7f815bc fixes your problem. The command in the README didn't quite match the hyperparameters in the paper. The correct command is:
python -u run_miniimagenet.py --shots 1 --inner-batch 10 --inner-iters 8 --meta-step 1 --meta-batch 5 --meta-iters 100000 --eval-batch 5 --eval-iters 50 --learning-rate 0.001 --meta-step-final 0 --train-shots 15 --checkpoint ckpt_m15t
Thank you very much! Will try the new command with complete data.
@unixpickle By the way, 7f815bc changes --eval-batch 15 --train-shots 5 to --eval-batch 5 --train-shots 15. I could understand the change of eval-batch, which behaves the same as reducing the learning rate (to 1/3 of original one) in the inner loop during the evaluation stage. However, I wonder why train-shots should be increased. In fact, it seems train-shots=1 matches the best when considering the similarity between the training and the evaluation stage.
@xwjabc for whatever reason, we found that training on more "shots" helped Reptile's performance, probably because it allows you to take more diverse gradient steps during each inner-loop. Table 4 of Appendix A in the paper specifies the hyper-parameters, this included.
@unixpickle
Got it. Thank you for the clarification!
@unixpickle @xwjabc I had the same problem with the incomplete miniimagenet data downloaded from fetch_data.sh,most of folder is empty images. Could you send me a complete data？My email address is lampardwk518@163.com ,thanks.
I used to have the dataset on my OpenAI machine and in the cloud. Unfortunately, I no longer have access to either copy. I'll see if I can find it sitting anywhere else, but I doubt I can.
What a pity! Thanks for your reply