Autolabeling custom 3D Objects

[happyCodingSusan]

I want use your method to label my own 3D objects (CAD models are available). What steps do i need to follow?

[Sergey Zakharov]

Hi Susan!
If you already have CAD models available and shape estimation is not really needed, then you only need to train a detector network (RCNN, DETR etc.) and a correspondence network (CSS net). The steps would be:

1. Train a detector to estimate bounding boxes containing your objects. If you don't have an annotated training set, you could render your CAD models, e.g. using BlenderProc;
2. Additionally render NOCS correspondences for your objects (similar to data/db/crops) e.g. using nocs-renderer;
3. Train a correspondence net to output NOCS from RGB (similar to train_css.py);
4. Estimate poses for detected objects based on estimated NOCS either by using PnP+RANSAC (if you have only RGB images available) or Kabsch+RANSAC (if you also have depth at hand). You can use our code for that;
5. Additionally you could apply refinement based on RGB/Depth similar to refine_css.py.

[happyCodingSusan]

@zakharos ,

Thank you very much for your kind and quick reply!
I tried the nocs-renderer, but encountered some runtime errors. I wonder if you can tell me all the packages and their versions in your renderer environment.
I set up the conda environment on windows 10. When I run "conda env create -n renderer -f env.yml" for the first time, "glumpy" cannot be installed successfully. After I changing python=3.5 in env.yml to python=3.6 and adding cython, I can create the environment. Then I encountered a runtime error at "expm" function. After reinstalling scipy of version 1.3.0, I can run the program and 24 images [ "synth_0000(~0007)_dpt.png, synth_0000(~0007)_dpt_vis.png, synth_0000(~0007)_img.png" ] were saved in the output folder. Howerer, it crashed at "self.program.draw(output, model.index_buffer)" in the function "draw_model". I guess there are still some package versions that do not match. I copied my conda renderer env packages and versions, as shown below, would you please send me your entire renderer environment packages and versions? Thank you very much for your help, Susan.

| packages | in environment | at C:\Anaconda\envs\renderer:

| Name | Version | Build Channel

blas | 1 | mkl | anaconda
certifi | 2020.12.5 | py36haa95532_0 |  
cffi | 1.14.4 | py36hcd4344a_0 |  
cudatoolkit | 10.1.243 | h74a9793_0 |  
cython | 0.29.21 | py36ha925a31_0 | anaconda
decorator | 4.0.11 | |  
freetype | 2.10.4 | hd328e21_0 |  
future | 0.16.0 | |  
glumpy | 1.1.0 | |  
icc_rt | 2019.0.0 | h0cc432a_1 | anaconda
intel-openmp | 2020.2 | 254 | anaconda
jpeg | 9b | hb83a4c4_2 |  
libpng | 1.6.37 | h2a8f88b_0 |  
libtiff | 4.1.0 | h56a325e_1 |  
lz4-c | 1.9.3 | h2bbff1b_0 |  
mkl | 2019.4 | 245 | anaconda
mkl-service | 2.3.0 | py36hb782905_0 | anaconda
mkl_fft | 1.2.0 | py36h45dec08_0 | anaconda
mkl_random | 1.1.0 | py36h675688f_0 | anaconda
networkx | 1.11 | |  
ninja | 1.10.2 | py36h6d14046_0 |  
numpy | 1.19.1 | py36h5510c5b_0 | anaconda
numpy-base | 1.19.1 | py36ha3acd2a_0 | anaconda
olefile | 0.46 | py_0 |  
opencv-python | 3.1.0.5 | |  
pillow | 8.1.0 | py36h4fa10fc_0 |  
pip | 20.2.4 | py36_0 | anaconda
plyfile | 0.5 | |  
pycparser | 2.2 | py_2 |  
pyglet | 1.2.4 | |  
pyopengl | 3.1.1a1 | py36_0 | anaconda
python | 3.6.12 | h5500b2f_2 | anaconda
pytorch | 1.4.0 | py3.6_cuda101_cudnn7_0 | pytorch
scipy | 1.3.0 | |  
setuptools | 50.3.0 | py36h9490d1a_1 | anaconda
six | 1.15.0 | py_0 | anaconda
sqlite | 3.33.0 | h2a8f88b_0 | anaconda
tk | 8.6.10 | he774522_0 |  
torchvision | 0.5.0 | py36_cu101 | pytorch
triangle | 20200424 | |  
vc | 14.1 | h0510ff6_4 | anaconda
vispy | 0.4.0 | |  
vs2015_runtime | 14.16.27012 | hf0eaf9b_3 | anaconda
wheel | 0.35.1 | py_0 | anaconda
wincertstore | 0.2 | py36h7fe50ca_0 | anaconda
xz | 5.2.5 | h62dcd97_0 |  
zlib | 1.2.11 | vc14h1cdd9ab_1 | [vc14] anaconda
zstd | 1.4.5 | h04227a9_0 |  

[Sergey Zakharov]

Hi Susan,

Sorry for the late reply. Unfortunately I didn't use the NOCS renderer on Windows and I currently don't have a machine to try... However, please create an issue in the respective repository and maybe some other users (or myself once I have access to a Windows machine) will be able to provide an environment for Windows. Thanks!