GPU required for training?

Question

GPU required for training?

flaime-ai opened this issue 4 years ago · comments

I am trying to train a new scene and am running into memory issues with training.

I have tried with a single Titan RTX (24GB) card, and a multi GPU setup (4 x Tesla T4, each 16GB)
With both I am receiving a CUDA out of memory error on the first epoch of training.

I was wondering which GPU set ups you used for training? As I would assume these should be sufficiently big to train the model.

Artem Sevastopolsky · Answer 1 · Tue Jul 14 2020 01:57:21 GMT+0800 (China Standard Time)

We used GeForce GTX 1080 Ti with less than 12 GB GPU memory, which should be more than enough for training in most cases. Not sure where your issue comes from; perhaps you can try decreasing batch size if it's more than 1 in your experiment (--batch_size parameter of train.py). Does it happen during running the example from the readme?

flaime-ai · Answer 2 · Tue Jul 14 2020 02:19:44 GMT+0800 (China Standard Time)

Thanks - Okay, there must be something else up.
I can get both the examples running with no problems.
I'll try decreasing the batch size.
Also, it may be to do with the point cloud generated from agisoft, I didn't optimise that.