Integrate Ha-Nerf in nerfstudio
dberga opened this issue · comments
First check in pipelines where "encode_a" and "use_mask" is applied
https://github.com/rover-xingyu/Ha-NeRF/blob/main/models/networks.py
https://github.com/rover-xingyu/Ha-NeRF/blob/main/train_mask_grid_sample.py#L36
https://github.com/rover-xingyu/Ha-NeRF/blob/main/eval.py#L131
https://github.com/rover-xingyu/Ha-NeRF/blob/main/hallucinate.py#L178
Then re-implement a new case (take vanilla nerf as example)? Replacing the embeddings encoding and implicit mask
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/models/vanilla_nerf.py
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/fields/vanilla_nerf_field.py
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/field_components/embedding.py
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/field_components/encodings.py
After including the model in https://github.com/dberga/nerfstudio/blob/main/nerfstudio/models, you must include the model instruction name for "ns-train model" in https://github.com/dberga/nerfstudio/blob/main/nerfstudio/configs/method_configs.py
Some specific lines from the official repo
overall definitions (XingYu Ha-Nerf class)
https://github.com/rover-xingyu/Ha-NeRF/blob/main/train_mask_grid_sample.py#L36
forward usage (training)
https://github.com/rover-xingyu/Ha-NeRF/blob/main/train_mask_grid_sample.py#L121
https://github.com/rover-xingyu/Ha-NeRF/blob/main/train_mask_grid_sample.py#L128
inference usage (eval)
https://github.com/rover-xingyu/Ha-NeRF/blob/main/eval.py#L147
rendering inference with appearance encoding
https://github.com/rover-xingyu/Ha-NeRF/blob/main/models/rendering.py#L112
Added Ha-Nerf model definitions and imported main modules from official repo
283511a
Taken Vanilla Nerf definition and added appearance field (using E_attr) and occlusion (implicit_mask).
See ha-nerf definition using vanilla-nerf structure and added ha-nerf fields / modules:
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/models/ha_nerf.py
See imported functions here:
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/field_components/hallucidated.py
See field definitions here:
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/fields/ha_nerf_field.py
See execution config (and default params, defined keyword for ns-train as "ha-nerf"):
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/configs/method_configs.py
Pending / to do:
-Remake field definitions (./nerfstudio/fields/ha_nerf_field.py)
-Update ./nerfstudio/models/ha_nerf.py usage