dazinovic / neural-rgbd-surface-reconstruction

Hi, @dazinovic
Thx for sharing your great work!
I'm confused about the meaning of sdf used in your code. I hope you can help me figure it out.

neural-rgbd-surface-reconstruction/losses.py

Lines 31 to 38 in 0178274

    
           def get_sdf_loss(z_vals, target_d, predicted_sdf, truncation, loss_type): 
        
               front_mask, sdf_mask, fs_weight, sdf_weight = get_masks(z_vals, target_d, truncation) 
        
               fs_loss = compute_loss(predicted_sdf * front_mask, tf.ones_like(predicted_sdf) * front_mask, loss_type) * fs_weight 
        
               sdf_loss = compute_loss((z_vals + predicted_sdf * truncation) * sdf_mask, target_d * sdf_mask, loss_type) * sdf_weight 
        
               return fs_loss, sdf_loss

As in this function, it seems that sdf is a ratio between the actual distance and the truncation value.
While, when it turns to this render function,

neural-rgbd-surface-reconstruction/optimize.py

Lines 172 to 183 in 0178274

    
           def sdf2weights(sdf): 
        
               weights = tf.math.sigmoid(sdf / truncation) * tf.math.sigmoid(-sdf / truncation) 
        
               signs = sdf[:, 1:] * sdf[:, :-1] 
        
               mask = tf.where(signs < 0.0, tf.ones_like(signs), tf.zeros_like(signs)) 
        
               inds = tf.math.argmax(mask, axis=1) 
        
               inds = inds[..., tf.newaxis] 
        
               z_min = tf.gather(z_vals, inds, axis=1, batch_dims=1) 
        
               mask = tf.where(z_vals < z_min + sc_factor * truncation, tf.ones_like(z_vals), tf.zeros_like(z_vals)) 
        
               weights = weights * mask 
        
               return weights / (tf.reduce_sum(weights, axis=-1, keepdims=True) + 1e-8)

it seems that sdf represents a distance in normalized space.
So, if we use the second meaning which is sdf is a distance in normalized space, should the sdf loss function change to this?

def get_sdf_loss(z_vals, target_d, predicted_sdf, truncation, loss_type):

    front_mask, sdf_mask, fs_weight, sdf_weight = get_masks(z_vals, target_d, truncation)

    fs_loss = compute_loss(predicted_sdf * front_mask, truncation * tf.ones_like(predicted_sdf) * front_mask, loss_type) * fs_weight
    sdf_loss = compute_loss((z_vals + predicted_sdf) * sdf_mask, target_d * sdf_mask, loss_type) * sdf_weight

    return fs_loss, sdf_loss

The predicted SDF is always normalized between -1 (behind the surface) and +1 (in front of the surface). In the loss, it needs to be multiplied by truncation since all the other variables are in metric space (possibly scaled by sc_factor from the config file).

Let's say the depth map pixel has a value of 0.8, the sample in z_vals has a value of 0.775 (difference of 2.5 cm). Then the predicted SDF needs to be 0.025 / truncation (0.5 with a truncation of 5 cm). So to turn this back into metric space and apply the loss, you need to multiple the predicted SDF by the truncation size.

Hi, Dazinovic,
Thanks for your reply. That makes sense.
But I still have a question about the weights in function sdf2weights.
Why do you use the ratio sdf/truncation as input? Intuitively, they're not in the same space.

Hi, Dazinovic,
Thanks for your reply. That makes sense.
But I still have a question about the weights in function sdf2weights.
Why do you use the ratio sdf/truncation as input? Intuitively, they're not in the same space.

The author only uses truncation without scfactor in sdf2weight, which is confusing.

Yes. The 'truncation' in sdf2weights function is more like a scale factor. As said in the paper, it controls how fast the weight falls to zero. It's not actually the truncation distance but happens to get a good result with the same value as "truncation distance". Those are only my suspects. Hopefully, the author will give a better comment about it.

Yes. The 'truncation' in sdf2weights function is more like a scale factor. As said in the paper, it controls how fast the weight falls to zero. It's not actually the truncation distance but happens to get a good result with the same value as "truncation distance". Those are only my suspects. Hopefully, the author will give a better comment about it.

You are right about that. We used a fixed scaling of 20.0 inside sdf2weight in our experiments (decided to use that after trying out a few numbers), but noticed last minute before submitting the paper that it can also be written more elegantly by using the truncation distance.

hi, @hirotong
Do you know the meaning of 'mask' in sdf2weights() ?

Do you know the meaning of 'mask' in sdf2weights() ?

A ray might pass through multiple surfaces. That's why the samples past the first truncation region are masked out.

I got it! Thanks for your reply.
There is another question:
Why do you use “signs = sdf[:, 1:] * sdf[:, :-1]” to generate mask instead of sdf directly?

Why do you use “signs = sdf[:, 1:] * sdf[:, :-1]” to generate mask instead of sdf directly?

No particular reason. I explicitly search for the zero-crossing (sign change). Any other implementation that produces the same result will work, too.

	def get_sdf_loss(z_vals, target_d, predicted_sdf, truncation, loss_type):

	front_mask, sdf_mask, fs_weight, sdf_weight = get_masks(z_vals, target_d, truncation)

	fs_loss = compute_loss(predicted_sdf * front_mask, tf.ones_like(predicted_sdf) * front_mask, loss_type) * fs_weight
	sdf_loss = compute_loss((z_vals + predicted_sdf * truncation) * sdf_mask, target_d * sdf_mask, loss_type) * sdf_weight

	return fs_loss, sdf_loss

	def sdf2weights(sdf):
	weights = tf.math.sigmoid(sdf / truncation) * tf.math.sigmoid(-sdf / truncation)

	signs = sdf[:, 1:] * sdf[:, :-1]
	mask = tf.where(signs < 0.0, tf.ones_like(signs), tf.zeros_like(signs))
	inds = tf.math.argmax(mask, axis=1)
	inds = inds[..., tf.newaxis]
	z_min = tf.gather(z_vals, inds, axis=1, batch_dims=1)
	mask = tf.where(z_vals < z_min + sc_factor * truncation, tf.ones_like(z_vals), tf.zeros_like(z_vals))

	weights = weights * mask
	return weights / (tf.reduce_sum(weights, axis=-1, keepdims=True) + 1e-8)

Concern about the meaning of sdf