apple / ml-neuman

Hello, thanks for the great work！
I have the following questions when processing the novel poses according to the instructions in #42:

If the goal is to make the human foot on the ground, why can't we just translate and rotate the body to the ground without changing the scale? Is the change to scale must be done?
What criteria should be used to determine the appropriate values for the translation, rotation, and scale? Can you give some practical guidance or tips for implementation?
Why is the human body much smaller than the background and seems to be in the wrong position when I use the same values in code to align them in Blender during the reproduction?

Yes, You have to the scale. It’s because we read the raw mocap data(AMASS), the scale of human in AMASS is very different from the scale from COLMAP. But I think it is possible to fix this by reading the scale from preprocessed data.
Tips: 1. Scale: You can use something in the point cloud as a reference, say a car/door/bike, then you roughly how to scale the human up to match that reference object. Rotation: You can import concatenate multiple human pcd(from different time steps) to a gigantic point cloud, this will allow you to use the whole motion sequence to align the feet. Translation: Define the camera trajectory first, render a sample bkg video, then put the human into the visible area, it will make sure the human is visible in the compositional rendering.

When you dumping the pcd of human, you wan to use the else brach , not the if branch:

ml-neuman/render_reposing.py

Lines 55 to 58 in 0149d25

    
           else: 
        
               manual_trans = np.array([0, 0, 0]) 
        
               manual_rot = np.array([0, 0, 0]) / 180 * np.pi 
        
               manual_scale = 1

	else:
	manual_trans = np.array([0, 0, 0])
	manual_rot = np.array([0, 0, 0]) / 180 * np.pi
	manual_scale = 1

Problem of alignment in Blender when processing new poses