Total training time on THUMOS14

Question

Total training time on THUMOS14

ivyvideo opened this issue 6 years ago · comments

Hi, I am running the code and find that it takes about two hours every 1000 iterations on Tesla M40, and I noticed that the default iteration is 60k totally...so it may cost so long time to finish training...
What is the reason for that? And what is your original training time?
Thanks!

huijuan88 · Answer 1 · Tue Aug 28 2018 22:59:42 GMT+0800 (China Standard Time)

If you resize the frames to be smaller, it would run faster. It also has a relationship with your machine. I am ~3 seconds per iteration. On Aug 28, 2018, at 05:45, ivyvideo <notifications@github.com<mailto:notifications@github.com>> wrote: Hi, I am running the code and find that it takes about two hours every 1000 iterations on , and I noticed that the default iteration is 60k totally...so it may cost so long time to finish training... What is the reason for that? And what is your origin training time? Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#39>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFOa_3Y3stiO76Vx_xuaGoIjpOlqOuMNks5uVRFWgaJpZM4WPRJX>.

ivyvideo · Answer 2 · Wed Aug 29 2018 15:51:07 GMT+0800 (China Standard Time)

Is it related to the setting: cfg.TRAIN.VIDEO_BATCH = 1? Maybe I should set this parameter larger? Like 32 or 64?

ivyvideo · Answer 3 · Wed Aug 29 2018 16:05:01 GMT+0800 (China Standard Time)

Sorry but I have another question... as shown below in the picture(part of minibatch.py), the parameter "fg_rois_per_video" is not used anymore after being assigned a value... so the network deals with 512 frames in only one video segment during forward process? And the batch_size is not used, either. Am I right?

huijuan88 · Answer 4 · Wed Aug 29 2018 22:12:41 GMT+0800 (China Standard Time)

Currently only one video in a batch. On Aug 29, 2018, at 04:05, ivyvideo <notifications@github.com<mailto:notifications@github.com>> wrote: Sorry but I have another question... as shown below in the picture(part of minibatch.py), the parameter "fg_rois_per_video" is not used anymore after being assigned a value... so the network deals with 512 frames in only one video segment during forward process? And the batch_size is not used, either. Am I right? [image]<https://user-images.githubusercontent.com/42105211/44774273-4592c400-aba5-11e8-8898-ea82b849af8a.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#39 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFOa_7cM4q9c4h44iXnhRitJUZ4NkPUnks5uVksvgaJpZM4WPRJX>.

ivyvideo · Answer 5 · Thu Aug 30 2018 10:15:40 GMT+0800 (China Standard Time)

Thank you so much for the kind reply.

ivyvideo · Answer 6 · Fri Aug 31 2018 09:30:51 GMT+0800 (China Standard Time)

@huijuan88 I have one more question...have you tested setting the "video_batch" more than 1?

huijuan88 · Answer 7 · Mon Sep 03 2018 01:41:10 GMT+0800 (China Standard Time)

No. That part is not implemented. On Aug 30, 2018, at 18:30, ivyvideo <notifications@github.com<mailto:notifications@github.com>> wrote: @huijuan88<https://github.com/huijuan88> I have one more question...have you tested setting the "video_batch" more than 1? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#39 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFOa_613oU-RQmzPC23Cf42c1zlVJgCMks5uWJHLgaJpZM4WPRJX>.

ivyvideo · Answer 8 · Mon Sep 03 2018 10:35:50 GMT+0800 (China Standard Time)

OK, your help is greatly appreciated~~