About training time for each 10 iterations
XiongweiWu opened this issue · comments
Thx for your work. I am reproducing your model with inception-style crop-scale and color augmentation as you claimed in #1 . Here I wonder what's your training time for each iteration with batchsize 256. With these data augmentation tricks my training time is ~18s for each 10 iterations, which is much slower than just random crop (4s).
I just tried this with 300K parameters on CIFAR10. it took nearly 2 hours to train for 3500 iterations!
and I'm on GTX1080. It also took 7+Gig of VRAM.
I have no idea how you guys are training ImageNet at this rate!
@Coderx7 CIFAR-10 is a very small and easy dataset and the experiments can usually finished within several hours with moderate models as VGG or res50 based on more powerful facility.
@shicai I just finish the ImageNet training and get 69.5% top-1 acc for 17 days with 2 P100 cards. I wonder whether u have faster caffe implementation since my data augmentation part is based on CPU.
@XiongweiWu You did'nt get my point. I'm implying this is taking like forever to train! at this rate training larger datasets such as ImageNet would take a huge amount of time!
and I have Corei7 4790K @4Ghz and GTX1080! yet I'm having alot of difficulties
@Coderx7 I suppose 1080ti is very fast with acceptable price. You can try this product which is much more powerful than original 1080 card.