how to use valid dataset to select a bestmodel to test?

[polarsun]

Dear,
Thank you for providing the excellent code, but I have some questions!
When I have finished the training process , I got 100 models, how I can select a best model from this list ?
I don't understand the eval.py how to select , it almost just to valid a lastest training model on valid dataset , not select best model, is it True?

[Atul Kumar]

For bigger (and distributed training) deep learning more like this, you can have two parallel process. One process train the model and save the model output at particular interval (like 10000 iteration etc). The other process check if there is new saved model file is there, if it's there then eval it and save the result. Also while evaluating you should make sure the training is not interrupted. If you have only one gpu then its good to first dump all the model at each 10000 iteration. And then run eval on all those model. if you have more than one gpu run eval on one gpu and training on another gpu. there is also an option of running eval on cpu if you want to run together.