Integrate Ref-Nerf in nerfstudio
dberga opened this issue · comments
Adding ref-nerf in nerfstudio has been already discussed here
nerfstudio-project#1469
Suggestions: i.e. take nerfacto or mip-nerf as example?
Note: the orientation calculation is already implemented in nerfacto (the Reflection Direction)
https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/models/nerfacto.py
https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/model_components/losses.py#L200
To do: the Directional encoding (Moses-Fischer vMF), but here we can utilize and modify the field processing, namely, the perceptron encoding (MLP) to generate that distribution instead:
https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/fields/nerfacto_field.py
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/fields/density_fields.py
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/fields/vanilla_nerf_field.py
https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/models/instant_ngp.py#L115
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/field_components/encodings.py
https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/field_components/mlp.py
After including the model in https://github.com/dberga/nerfstudio/blob/main/nerfstudio/models, you must include the model instruction name for "ns-train model" in https://github.com/dberga/nerfstudio/blob/main/nerfstudio/configs/method_configs.py
Done in
831b455
Check model definition in
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/models/refnerfacto.py
and its config (default params) in
https://github.com/dberga/nerfstudio/blob/main/nerfstudio/configs/method_configs.py
Pending: test usage
@dberga have you tested your implementation on datasets provided in paper?