Train TexPose on custom objects
EvdoTheo opened this issue · comments
Hello @HanzhiC, your work is excellent. I'm trying to train TexPose on custom objects, so first, I would like you to clarify the input requirements of the network in order to conduct the data preparation process.
Second, aside from the training part, I Would like to ask you, if you have executed the pose estimation algorithm in real-time scenarios, if not, do you believe that is plausible?
Thank you for the interests!
To train on custom objects: you need to prepare a set of synthetic images with pose labels and a set of color images in real domain. To render synthetic data, we recommand using BlenderProc2, and organize everything in BOP format.
If you have some reasonable GPU, yes, nowadays SotA pose estimators can be real-time, e.g., GDR-Net.
Sorry for the late reply, but I have some questions. Do I skip the Compute box for ray sampling and the Generate surfel information and synthetic image steps or do I have to complete them in order to continue in the training process?
- Computing box is a must as training NeRF requires near and far bounds for ray sampling.
- Surfel information is computed to improve the learned texture's accuracy, while without it you can still gain SotA performance, as illustrated in our main paper.
So then, I follow the given scripts? The steps I've completed until now are:
- Export the CAD file in ply format
- Modify my custom dataset in bop format (RGB, depth, mask,mask_visib and the JSON files) and
- Split the dataset into training and testing samples (train.txt and test.txt)
Are there any other conditions I need to satisfy in order to train my dataset?
Yes. Also, don't forget to pre-train your pose estimator with synthetic PBR data.
For the pre-training of the pose estimator, do I follow the Self6d++ repository steps? Also, could you please clarify the meaning of the folders inside the lm_models directory (models,models_eval, models_xyz)?
Yes, please follow their instructions. Model_eval is for evaluation, model is for rendering. There are no big difference between them.