Giters

unibas-gravis / parametric-face-image-generator

Generate fully parametric face images from the Basel Face Model 2017

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Need a little help with the PCA basis reconstruction

Vargeel opened this issue · comments

commented

Hi,

I have been using the Basel face model to generate faces (specifically model2017-1_face12_nomouth.h5) and I would like to do some statistical study on the faces being generated. I built some code that generate a new face using the PCA basis and thought I would compare the face generated using this scala based generator.

Sadly I am not very well versed in scala or java and I must admit I am a bit lost in the code at the moment.
In python I used the values in shape_PCA_Basis=h5file['shape']['model'].get('pcaBasis') which is a matrix of shape (85764, 199).
To create a face I use the mean position of the vertices given in shape_average=h5file['shape']['model'].get('mean'), generate a (199,1) vector of random values (vector_rnd) and get the final position of vertices by doing : Vertices = np.reshape(shape_average+ np.matmul(shape_PCA_Basis,vector_rnd), (28588,3))

This seems to work well but if instead of using a random vector I us the one created in the .rps file this scala code creates I see big differences in the faces generated using both language. I render the face in Python using a pinhole camera which also distorts the final image but I think the changes in shape are way more limited in my code than they seem to be in the rendered scala image.

Did I misunderstand how to rebuild the face vertices?

If not could you please direct me to a way to output the vertices position of the face in the scala model? I assume I would have to change the scalismo-faces code as well.

Thank you in advance,
Vargeel

commented

Hi Vargeel,

can you add some examples images?
So far what you write makes sense - so the images could be very helpful to get an idea in which direction to search for the error.
Perhaps also render instances with [1, 0, 0, .....] as parameters for testing...
for the scala version you can easily generate them by the Basel Face Model viewer which allows to read an rps file
https://github.com/unibas-gravis/basel-face-model-viewer
We also use a pinhole camera - that should not be the issue.
Best
Bernhard

commented
commented

Hi Andreas,

figure_1

Here is a picture of my render in python on the left and in scala on the right. The spherical harmonics on the left are a constant since I am also having trouble extracting them from the rps file.

I am only rendering pixels that have a Vertice behind them so it has gaps where the vertices do not align directly.

As you can see the face reconstructed through Python is much closer to the mean face.

here are my intrensics :
camera_model = np.asarray([[2000.0, 0.0, 0.0], [ 0.0, 2000.0, 0.0], [ 0.0, 0.0, 1.0]])

The RPS file is attached as well as the vertices mean position, the shape descriptor extracted and the shape difference due to the vector.
for_github.tar.gz

commented

It is hard to tell what comes from the illumination/albedo and which part is actually shape. Can you keep the albedo/color parameters fixed to the mean when renderning? That my help to identify the problem. In case of the generator just set the parameters for color to 0 and it will generate the color mean with only shape variation.

You might also try to do the scaling with the eigenvalues according to Andreas post.

Best
Bernhard

commented

Hi Andreas and Bernhard,

Sorry I think I missed one your messages yesterday.

you have not scaled the pcaBasis correctly. You need to scale each vector with the corresponding pcaVariance

I assumed hat might be the case and tried that but the results were extremely distorted at the time. Can I just double check the math with you? When you say I need to use the PCAVariance to scale the vectors, is it something like this?
shape_PCA_Basis = hf['shape']['model'].get('pcaBasis').value shape = (85764, 199)
shape_PCA_Variance = hf['shape']['model'].get('pcaVariance').value shape = (199,)
rescaled_PCA_Basis = shape_PCA_Basis*shape_PCA_Variance shape = (85764, 199)

I can then use the shape vector alpha shape (,199) from the reference rps file.

alpha = alpha * Es_var_np
Vertices = shape_average+np.matmul(shape_PCA_Basis,alpha)

The result is quite strange though. The change in shape are massive now.
figure_1-1

for_github.tar.gz

Do you know why that may be?

commented

Also thanks for the advice Bernhard,

I tried generating images with the albedo components set to 0.

Here is a result :
figure_1-2

commented

I also redid the scaling with a normalized scale :
alpha = alpha*shape_PCA_Variance)/199.

and obtained the following result (mean face on the left, normalized changes in the middle and scala rendering on the right)

figure_1-3

I can see changes for sure but htey are still marginal compared to the scala rendering (the albedo is set to the mean)

commented
commented

Hi Andreas,

Since I was getting changes in shape that were about two orders of magnitude higher than expected I assumed there might have been a mean variance factor missing (199 was the size of the variance vector).

I applied the change you proposed :
alpha = alpha * np.sqrt(shape_PCA_Variance)

the results look more similar to the scala representation (at least in terms of pure physiognomy).

figure_1-4

I think the other disparities are due to the other components I am having trouble tuning so the images can be compared pixel wise which are the illumination parameters and the camera intrinsic parameters.

I was wondering about the part of the rps file generated under ["environmentMap"]["coefficients"].
Those seem to b Spherical Harmonics parameters but the images I get when using those directly are much darker than the ones in scala.
I was wondering if this might be because of the other lighting parameters (under ["directionalLight"]).
Are those representations of additional lights or the same light reference as the Spherical Harmonics coefficients?
figure_1-5

Additionally, @BernhardEgger mentioned yesterday that you use a pinhole camera in the renderer in scala. Might I ask if there is a way to output what the camera intrinsic matrix is at run time? I might have been mistaken but trying to reconstruct it from the parameters given in the rps file seemed ot provide strange results.

Once again thank you so much for your time and patience.

commented

I have managed to solve the spherical harmonics issue I was having. Turns out in the scalismo faces code there is a lambert kernel that scales the spherical harmonics after they have been set.
The weights are located at https://github.com/unibas-gravis/scalismo-faces/blob/9a00c5967349eb7861966cefd7f831d536d8822a/src/main/scala/scalismo/faces/parameters/Illumination.scala
line 96 and are applied in
https://github.com/unibas-gravis/scalismo-faces/blob/07a60366a66bc84592d4e51c88fa00f384601167/src/main/scala/scalismo/faces/render/PixelShaders.scala
line 162 for anyone that runs into the same issue.

Here are the current results.
figure_1-8

The last part required is to figure out the right intrensic parameter sin python for hte camera models to match.

commented

Let's close this one. If there is more to discuss please use:
https://groups.google.com/forum/#!forum/scalismo-faces