Rigid Transformations
HyunsooCha opened this issue · comments
Hi,
I am trying to run 'real_to_nerf.py' to use my research. I am implementing a face-tracking code using DECA. However, DECA uses camera parameters as (tx, ty, scale). I guess NerFACE uses (tx, ty, tz) for rigid transformations. So, I would like to know how to change the DECA camera parameters to rigid transformations used in NerFACE.
The purpose of my question is to make text files - intrinsics.txt, expression.txt, and rigid.txt
I appreciate your kindness. Your research is helpful to me in understanding the area of study.
How is your progress and whether you can complete the effect in the paper? I'm stuck at this step
I used DECA, which can be found on GitHub, as a face tracker. However, DECA's output - pose, expression, and shape tensors do not fit NerFACE. I also tried to apply Expression Net, which is another 3DMM. Unfortunately, this network also does not perfectly fit NerFACE. I think only Face2Face can make three text files.
And, as you maybe already know, pose and expression tensors are pretty sensitive at a scale of 0.01. So, I guess only Face2Face can make exact output for NerFACE. I hope the author provides the code.
@stephencha It should in principle be able to work with any face tracker (just have to adjust the dimensionality of the expression vector). The authors of IM Avatars from ETH managed to run it with their own tracker (not sure which, but I assume it is based on FLAME).
@yc4ny You are essentially asking how to implement face 3DMM tracking. This is too big of a subject to explain here. The details of our exact face tracker can be found in Face2Face [Thies et al]. We use this as an off-the-shelf preprocessing method.
I appreciate your reply. Happy day.
@stephencha It should in principle be able to work with any face tracker (just have to adjust the dimensionality of the expression vector). The authors of IM Avatars from ETH managed to run it with their own tracker (not sure which, but I assume it is based on FLAME).
@yc4ny You are essentially asking how to implement face 3DMM tracking. This is too big of a subject to explain here. The details of our exact face tracker can be found in Face2Face [Thies et al]. We use this as an off-the-shelf preprocessing method.
Hi, I'm also trying to use my own tracker. But the learned model simply ignored my expression input. Is there any special requirement or pre-process that has to be carried out on the expression coefficients? I saw that in this line the expression coefficients are divided by three and not sure why.
@stephencha It should in principle be able to work with any face tracker (just have to adjust the dimensionality of the expression vector). The authors of IM Avatars from ETH managed to run it with their own tracker (not sure which, but I assume it is based on FLAME).
@yc4ny You are essentially asking how to implement face 3DMM tracking. This is too big of a subject to explain here. The details of our exact face tracker can be found in Face2Face [Thies et al]. We use this as an off-the-shelf preprocessing method.Hi, I'm also trying to use my own tracker. But the learned model simply ignored my expression input. Is there any special requirement or pre-process that has to be carried out on the expression coefficients? I saw that in this line the expression coefficients are divided by three and not sure why.
The division by 3 is quite meaningless (an attempt to "normalize" the expressions).
in the evaluation script there is a line that freezes the expression. Change it to None
I have already set the ablate=None.
I think I found the problem, so in the train folder, the images seem out of order, but I ran my tracker on the person_1_train.mp4, thus the expression parameters are not corresponding to the frames. Do you remember the order of the images in the train folder? Thanks
Never mind, I found the order in index.npy. Thanks for your time.
@stephencha Hi, sorry to bother you. Have you got the way to produce the transformation matrix in the json file and the focal etc ? I hope to apply this method to my own dataset but I have no idea how to get them. The tool I am using is from https://github.com/philgras/video-head-tracker, but they seems to have a different coordinate.
@Jianf-Wang
hey bro, have you successfully bridge the tracking preprocessing requirement with this repo https://github.com/philgras/video-head-tracker ?