many questions about training
Jaywxy opened this issue · comments
I git clone your code files ，and train on 4*3090 ，But I found that the CPU usage is very low，And the gpu usage is also very jumpy, the output shows that the training takes roughly two and a half days to complete, do I think there is something wrong with the code? Because I have tested, the server‘s cpu is not a problem, my other models, only use a gpu , the CPU utilization is higher than the current utilization, and the GPU is not so big fluctuations, I hope you can give me a reply, thank you!
It's ok. When I do experiments, the time are also 3 days with 50 epochs on SemanticKITTI. You can do experiment with SemanticPOSS dataset, or don't use all sequences of SemanticKITTI. And I didn't pay much attention to the use of CPU. I noticed the GPU of fluctuations, but I don't know the reason, maybe I use the dataparellel training.
ok,Thank you for your answer. Maybe the code needs to be optimized somewhere.thank you for your work！
但是我之前使用23090进行训练了模型，进行infer时只能跑到13FPS左右？这可能是什么原因呢？
But I used 23090 to train the model before, but when I infer, it can only run around 13FPS? What could be the reason for this?
26FPS的模型是我使用半浮点精度训练而来的模型，这样的模型运行速度会快很多。但是它训练时只能用一张卡，会导致训练速度大大下降，所以我后面整理代码时没有保存这部分代码。
请问使用半浮点数精度训练能达到文章64.5的结果吗？
请问使用半浮点数精度训练能达到文章64.5的结果吗？
Hello, I have also run the author's code. May I ask if you have achieve the MIOU of the article? Do you need to use a pretrained model to reproduce the author's results? Is it possible to train from scratch？
请问使用半浮点数精度训练能达到文章64.5的结果吗？
Hello, I have also run the author's code. May I ask if you have achieve the MIOU of the article? Do you need to use a pretrained model to reproduce the author's results? Is it possible to train from scratch？
半浮点精度训练也是提高训练精度的一种技巧，但是它训练时间太长，所以我没有用它跑完整的模型。在使用512x64的大小训练时，两种方法的精度没有多少差别
请问使用半浮点数精度训练能达到文章64.5的结果吗？
Hello, I have also run the author's code. May I ask if you have achieve the MIOU of the article? Do you need to use a pretrained model to reproduce the author's results? Is it possible to train from scratch？
半浮点精度训练也是提高训练精度的一种技巧，但是它训练时间太长，所以我没有用它跑完整的模型。在使用512x64的大小训练时，两种方法的精度没有多少差别
Wow, thank you for your reply. May I ask if the default input size in the code you provided is 512 × 64. Where can I adjust the input size？How did you train with input sizes of 64 × 1024 and 64 × 2048, and are there any additional operations？
查看训练命令中指定的配置文件 -ac config/arch/LENet.yaml，其中定义了许多训练使用的参数。input-size也在其中
查看训练命令中指定的配置文件 -ac config/arch/LENet.yaml，其中定义了许多训练使用的参数。input-size也在其中
Thank you for your reply, I understand now. So, in order to obtain the final result of 64 × 2048, do we need to train on 64 × 512 first. Is it correct to use a pretrained model of 64 × 512 to train 64 × 1024 and 64 × 2048？
直接训练就可以了。64 × 2048训练处理的精度是最高的