apple / ml-neuman

Official repository of NeuMan: Neural Human Radiance Field from a Single Video (ECCV 2022)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Some questions about reproduction on new datasets

hongzhengdong opened this issue · comments

Dear author Jiang,
I have some questions to run your code on my custom dataset.
Q1: My dataset is a video of moving person captured by a moving camera, which is similar with your datasets. However, the difference is that I am planning to use altogether 360 frames for training, which will be more than yours(~100frames), will it increase training time( as I don't know in the 7 days training time as you mentioned, which one is slower(bottomneck), Human Model or the Scene Model? Moreover, the movement of the human body is not a continuous movement, but divided into two different movements, will it affect the training of Human Model? Should I modify something?
Q2: As you mentioned, the 2 stages costs about 7 days for training on a V100, Can you tell me What is the most time-consuming operation in Stage1 and Stage2, seperately? Because the second stage cannot support full-GPU, I wonder if I can improve the training speed by easily modifying some configurations, so that I can see a relatively correct result in a shorter time.

1.1 will it increase training time? I don't think it will increase the training time, as long as there is enough coverage of the human body.
1.2 which one is slower(bottleneck), Human Model or the Scene Model? The human model takes more time to train, in our case, 2 days for the scene model, and 5 days for the human model.
1.3 the movement of the human body is not a continuous movement but divided into two different movements. I think it should be fine, we don't apply any temporal smoothness constraints.
2. What is the most time-consuming operation in Stage1 and Stage2, separately? Stage 1 is to train a vanilla nerf model, GPU can be fully utilized in stage 1. Stage 2 takes more time, because finding the closest point on a mesh runs on CPU. I think NeuralActor should have a CUDA implementation of this function, you may want to have a look. https://github.com/lingjie0206/Neural_Actor_Main_Code/blob/master/fairnr/clib/include/point_mesh.h