About the training time

Question

About the training time

liuyang-ict opened this issue 2 years ago · comments

Yang Liu commented 2 years ago

How much V100 GPU-time when TubeDETR was trained on HC-STVG?

Antoine Yang · Answer 1 · Wed Apr 13 2022 19:25:02 GMT+0800 (China Standard Time)

20 epochs on HC-STVGv2 took about 7 hours on 16 V100 for the best configuration (stride 4 resolution 352).

Yang Liu · Answer 2 · Wed Apr 13 2022 20:41:42 GMT+0800 (China Standard Time)

Thanks a lot~~

Peng Zheng · Answer 3 · Sun Jun 19 2022 23:13:39 GMT+0800 (China Standard Time)

If fewer GPUs are used for training, like 2~8 V100, will the accuracy decrease?
By the way, are your V100 cards all 32GB? Thanks in advance.

Peng Zheng · Answer 4 · Sun Jun 19 2022 23:19:41 GMT+0800 (China Standard Time)

By the way again, I think the preproc_hstvgv2.py in preproc should be preproc_hcstvgv2.py? :)

Antoine Yang · Answer 5 · Mon Jun 20 2022 03:21:18 GMT+0800 (China Standard Time)

I haven't tried training with a lower number of V100 but I don't think this would decrease the results. All the V100 used in experiments are 32GB, but you can use 16GB V100 with a lower resolution and/or higher temporal stride. The name of the preproc_hstvgv2.py file indeed had a typo, thanks for catching this!