Is it possible to extract meshes from the learned volume?

[SunshineAtNoon]

Hi, I was wondering if we can extract meshes from the learned NeRFs as in the original NeRF repo. Specifically, how to sample a valid cube? What's the range of the coordinates should be? Thanks!

[Guy Gafni]

Hi, I think it should be possible, I haven't tried it though.

The whole scene is bounded in the unit cube, scene is normalized such that the face is at an average of z=0.5 from the camera.

A different way to get a mesh that should be ok for faces, without having to samples a grid and do MC, as faces are very simple surfaces, is to render an expected depth map (use more ray samples though), unproject it to a point cloud and run poison reconstruction.

[SunshineAtNoon]

Hi, I think it should be possible, I haven't tried it though.

The whole scene is bounded in the unit cube, scene is normalized such that the face is at an average of z=0.5 from the camera.

A different way to get a mesh that should be ok for faces, without having to samples a grid and do MC, as faces are very simple surfaces, is to render an expected depth map (use more ray samples though), unproject it to a point cloud and run poison reconstruction.

Thanks so much for your response. Could you point me to which code you use to extract the mesh? Thanks!

[Guy Gafni]

The meshification in this image is done in Meshlab, screened Poisson surface reconstruction.

To backproject a depth image from our code, you can use the following code within the evaluation script:
Here, depth should be a tensor of size H,W,1

def unproject_torch(depth, focal, world_coords=True,pose=None, name=None, H=512,W=512,save=True):
    raster = meshgrid_xy(torch.arange(W), torch.arange(H))
    u = raster[0]
    v = raster[1]
    x = (u - focal[2]) * depth.cpu() / ((focal[0]))
    y = (v - focal[3]) * depth.cpu() / ((focal[1]))
    z = depth.cpu()
    pts = torch.dstack((x, y, z)).reshape(-1,3)
    dims=3
    if world_coords and pose is not None:
        points_hom = torch.cat((pts, torch.ones(pts.shape[0], 1)), dim=1)#.to(depth.device)
        pts = torch.matmul(torch.inverse(pose).cpu(), points_hom.transpose(1, 0)).transpose(0, 1)
        dims=4
    save_pc(pts,name,dims=dims)
    return pts

def save_pc(tensor,name=None,dims=4):
    from pathlib import Path
    if name is None:
        from datetime import datetime
        now = datetime.now()  # current date and time
        date_time = now.strftime("%m_%d_%Y_%H:%M:%S")

    fname = name if name is not None else date_time + ".obj"

    print("saving point cloud %s" % fname)

    Path(fname).write_text("\n".join([f"v {p[0]} {p[1]} {p[2]}" for p in tensor.reshape(-1,dims)]))
    return

To generate a depth map, you'd have to use the rendering weights to get a weighted sum of the z values of the ray points. in volume_render_radiance_field, use

    depth_map = weights * depth_values
    depth_map = depth_map.sum(dim=-1)

[SunshineAtNoon]

That's great! Thanks so much for your help!