Difference between generic FLAME models and VOCA-compatible FLAME models?
arcgt opened this issue · comments
What is the difference between generic FLAME model meshes and VOCA-compatible FLAME model meshes?
For example, running demo.py using the provided FLAME_sample.ply works perfectly. Similarly, using TF_FLAME's 'generate VOCA template' also produces random face shape meshes that work perfectly.
However, a mesh generated (in FLAME topology) using TF_FLAME's 'fit registered 3D meshes' produces an output animation that has the rendered model mesh displayed, but only distortions to the mouth and various face regions (but no comprehensible lip sync motions).
I initially thought this was due to the mesh not being in 'zero pose', however a zero pose FLAME topology mesh I have generated using FLAME's Blender add-on also has the same distortions when used as the template mesh in demo.py. Similarly, meshes aligned using ICP (Iterative Closest Point) to be in zero pose have the same distortions.
Is there a way to convert an arbitrary model in FLAME topology to a VOCA-compatible template mesh?
Thanks a lot and great work :)
[Edit] I've just tried importing and exporting the default VOCA-compatible template model from Blender to use as the template mesh in demo.py as a sanity check, however this creates the same artefacts, so perhaps exporting from Blender is what is causing the problem. Is there any reason why this could be?
For VOCA, only the mesh topology matters and this is same for all FLAME models. Generating templates by randomly sampling the identity face space should work for all FLAME models. Only the eye blink demo works solely for FLAME2019, as it uses pre-computed expression parameters to edit the eyelids.
When fitting any FLAME model to a registered 3D mesh in FLAME topology with the TF_FLAME code, global rotation, pose (head pose, jaw pose) and expression parameters must be set to zero. Then you should be able to use the result as VOCA template.
If you considered setting all these parameters to zero before exporting the mesh, and the problem remains, can you please show some renderings of these errors?
Thank you for the reply!
As mentioned in the edit of my original pose, I think the problem may actually be with exporting the file from Blender.
I am working with OBJ FLAME files, and need to convert to PLY to input into VOCA.
For this, I need to import the OBJ file and export as PLY from Blender. It seems this step might be causing the artefacts I mentioned, as using the FLAME template mesh provided in the demo, exporting this as an OBJ file and back to a PLY file in Blender (without any other changes), then inputting back to VOCA creates the same artefacts (where the original template worked perfectly). I've attached a video of the artefacts described:
video.mp4
Do you know if it may be some export settings in Blender causing this, or conversion from OBJ to PLY?
I see, it seems your vertex order is somehow scrambled. Make sure that Blender exports the meshes without changing the vertex order. (This could be either when importing OBJ into Blender, which has an "keep vertex order" flag that must be checked, or when exporting the mesh)
I see, thank you! I will try this now and let you know if it works!
As a sanity check you can also take the OBJ and convert it to PLY with MeshLab, which does not change vertex orders by default.
I was able to get it to work using MeshLab, exporting the original working template as OBJ, then back to PLY, to generate a working animation output.
I will now try with TF_FLAME 'fit to 3D mesh' and setting global pose to zero.
How can I set the pose to zero after fitting FLAME to a 3D mesh in FLAME topology (or does the mesh originally need to be in zero-pose)?
Zero out the parameters for global rotation and translation, jaw, head, and eyeball poses, and the expression vector after the fitting, basically keeping only the identity shape parameters non-zero. Then just output the result of the forward pass through the FLAME layer for these parameters. This will give you a mesh in "zero pose"
Possibly a stupid question, but is there a simple way of setting all the parameters to zero? A specific file I could modify or run? I had a brief look but could not find anything from TF_FLAME. Thank you in advance!
Managed to make everything work. I extracted all the translation, rotation, pose, shape and expression parameters from "fit_3D_mesh.py", then generated a neutral mesh using only the extracted shape parameter with all other parameters zero-ed out using the assign method in "sample_FLAME.py".