Question about the Equation(1) implementation

[Cai Minghong]

Hi, Thanks for your awesome work, the result is very cool!
But I am not fully understanding the density implementation of Equation(1) in the sugar_model.py.

The code is here:

shift = (x[:, None] - closest_gaussian_centers)
warped_shift = closest_gaussian_inv_scaled_rotation.transpose(-1, -2) @ shift[..., None]
neighbor_opacities = (warped_shift[..., 0] * warped_shift[..., 0]).sum(dim=-1).clamp(min=0., max=1e8)
neighbor_opacities = density_factor * closest_gaussian_strengths[..., 0] * torch.exp(-1. / 2 * neighbor_opacities)
densities = neighbor_opacities.sum(dim=-1)

I think closest_gaussian_inv_scaled_rotation.transpose(-1, -2) is corresponding to the inverse of the covariance matrix.
There are two points that make me confused.

1. warped_shift[..., 0] * warped_shift[..., 0] seems like the element of closest_gaussian_inv_scaled_rotation is calculated twice, not fully corresponding to the equation?
2. why the closest_gaussian_strengths[..., 0] exists? I do not fully understand the role of it.

Could you provide any help? Thank you very much.

[Antoine GUEDON]

Hello @onevfall,

Thank you for your nice words!

Here are some answers for you:

Question 1. The covariance matrix can be computed as $\Sigma_g=R_g S_g S_g^T R_g^T$. Let's define the shift $w=(x - \mu_g)$.
To compute the density of the point, you first need to compute $w^T \Sigma_g^{-1} w$. You can actually rewrite this computation (which is a scalar product) as:

$w^T \Sigma_g^{-1} w = w^T (R_g S_g S_g^T R_g^T)^{-1} w$
$w^T \Sigma_g^{-1} w = w^T R_g S_g^{-1} S_g^{-T} R_g^T w$
$w^T \Sigma_g^{-1} w = || S_g^{-T} R_g^T w ||_2^2$

So you can compute this quantity as the squared norm of the shift $w$ multiplied by the transpose of the matrix $R_g S_g^{-1}$. I call this matrix closest_gaussian_inv_scaled_rotation in the code.
After multiplying the shift $w$ by this matrix, I just have to take the norm of the resulting "warped" shift, which is why I finally use (warped_shift[..., 0] * warped_shift[..., 0]).sum(dim=-1).

Question 2. The tensor closest_gaussian_strengths[..., 0] just contains the opacities $\alpha_g$ of the Gaussians, which are required in the formula. density_factor is just a constant factor that can be used to adjust or play with the densities to avoid numerical instability in the computation.

I hope this answer will be helpful to you!

[Cai Minghong]

Wow, I understand it fully. Thanks very much for your kind help! @Anttwo