GCABC123 / magnetron-GET3D

✭ MAGNETRON ™ ✭: for making IMAGINATION PROXIA.

magnetron.artificial-intelligence-2.0.mincloud.proxia--IMAGINATION-A2

🤖 THE ABC 123 GROUP ™ 🤖

🌐 GENERAL CONSULTING ABC 123 BY OSAROPRIME ™.

🌐 ABC 123 USA ™

🌐 ABC 123 DESYGN ™

🌐 ABC 123 FILMS ™

=============================================================

                 🌐 𝗠𝗔𝗚𝗡𝗘𝗧𝗥𝗢𝗡 ™ 🌐

🌐 𝗔𝗥𝗧𝗜𝗙𝗜𝗖𝗜𝗔𝗟 𝗜𝗡𝗧𝗘𝗟𝗟𝗜𝗚𝗘𝗡𝗖𝗘 𝟮.𝟬 ™ : IMAGINATION PROXIA (A-2)

For making an IMAGINATION PROXIA that generates a 3D representation of of an object detected in an IMAGE.

*️⃣📶🤖

+++++++++++++++++++++++++++++++

ASTRAL BODY MINDCLOUD:

PRANIC BODY MINDCLOUD:

INSTINCTIVE MIND MINDCLOUD:

ASTRAL MIND MINDCLOUD: ✅ (IMAGINATION PROXIA A-2)

PRANIC MIND MINDCLOUD:

++++++++++++++++++++++++++++=++

REQUIREMENTS:

[*] Software Requirements: Python, Pytorch

[*] HARDWARE REQUIREMENTS: fast TPU/GPU (Tensor or Graphics Processing Unit)

[*] DEPENDENCIES: [DOCKERFILE INCLUDED]

=============================================================

USAGE:

e.g On an ASTRAL MINDCLOUD this PROXIA can be used to process INFORMATION sent to it from an INSTINCTIVE MIND PROXIA/MINDCLOUD (OBJECT DETECTION). So for example if the ROBOT encounters an intersting photograph or it sees a location from a particular angle it can take a photograph and then reconstruct the scene virtually from the photograph to better understand the layout of the environment or even what it would be like for a human to be there. ARTIFICIAL INTELLIGENCE 2.0 ™ ROBOTS can synthesize new views of places they cannot access but can see from particular angles (very useful).

e.g To see how an object would like in different lighting situations.

Prerequisite reading:

🌐 ARTIFICIAL INTELLIGENCE PRIMER ™: https://www.facebook.com/artificialintelligenceprimer

🌐 ARTIFICIAL INTELLIGENCE 2.0 ™ DOCUMENTATION: https://www.facebook.com/aibyabc123/

🌐 MEMBER'S CLUB ™: https://www.facebook.com/abc123membersclub/

👑

INCLUDED STICKERS/SIGN:

FIND STICKERS HERE: https://bit.ly/3B8D3lE

• PROMOTIONAL MATERIAL FOR 𝗠𝗔𝗚𝗡𝗘𝗧𝗥𝗢𝗡 𝗧𝗘𝗖𝗛𝗡𝗢𝗟𝗢𝗚𝗬 ™. (CUSTOM GRAPHICS BY 𝗔𝗕𝗖 𝟭𝟮𝟯 𝗗𝗘𝗦𝗬𝗚𝗡 ™/𝗢𝗦𝗔𝗥𝗢 𝗛𝗔𝗥𝗥𝗜𝗢𝗧𝗧). THE 𝗠𝗔𝗚𝗡𝗘𝗧𝗥𝗢𝗡 𝗧𝗘𝗖𝗛𝗡𝗢𝗟𝗢𝗚𝗬 ™ SYMBOL/LOGO IS A TRADEMARK OF 𝗧𝗛𝗘 𝗔𝗕𝗖 𝟭𝟮𝟯 𝗚𝗥𝗢𝗨𝗣 ™ FOR 𝗠𝗔𝗚𝗡𝗘𝗧𝗥𝗢𝗡 𝗧𝗘𝗖𝗛𝗡𝗢𝗟𝗢𝗚𝗬 ™. 𝗧𝗛𝗘 𝗔𝗕𝗖 𝟭𝟮𝟯 𝗚𝗥𝗢𝗨𝗣 ™ SYMBOL/LOGO IS A TRADEMARK OF 𝗧𝗛𝗘 𝗔𝗕𝗖 𝟭𝟮𝟯 𝗚𝗥𝗢𝗨𝗣 ™.

*️⃣📶🤖

• PROMOTIONAL MATERIAL FOR 𝗔𝗥𝗧𝗜𝗙𝗜𝗖𝗜𝗔𝗟 𝗜𝗡𝗧𝗘𝗟𝗟𝗜𝗚𝗘𝗡𝗖𝗘 𝟮.𝟬 ™. (CUSTOM GRAPHICS BY 𝗔𝗕𝗖 𝟭𝟮𝟯 𝗗𝗘𝗦𝗬𝗚𝗡 ™/𝗢𝗦𝗔𝗥𝗢 𝗛𝗔𝗥𝗥𝗜𝗢𝗧𝗧) THE 𝗗𝗥𝗔𝗚𝗢𝗡 & 𝗖𝗥𝗢𝗪𝗡 👑 SYMBOL/LOGO IS A TRADEMARK OF 𝗧𝗛𝗘 𝗔𝗕𝗖 𝟭𝟮𝟯 𝗚𝗥𝗢𝗨𝗣 ™ ASSOCIATED WITH TECHNOLOGY. 𝗧𝗛𝗘 𝗔𝗕𝗖 𝟭𝟮𝟯 𝗚𝗥𝗢𝗨𝗣 ™ SYMBOL/LOGO IS A TRADEMARK OF 𝗧𝗛𝗘 𝗔𝗕𝗖 𝟭𝟮𝟯 𝗚𝗥𝗢𝗨𝗣 ™.

You must display the included stickers/signs (so that it is clearly visible) if you are working with MAGNETRON ™ TECHNOLOGY for the purposes of determining whether you want to purchase a technology license or not. This includes but is not limited to public technology displays, trade shows, technology expos, media appearances, Investor events, Computers (exterior), MINDCLOUD STORAGE (e.g server room doors, render farm room doors) etc.

🌐 NOTE: IMAGINATION PROXIA A IS DESCRIBED IN THE 𝗔𝗥𝗧𝗜𝗙𝗜𝗖𝗜𝗔𝗟 𝗜𝗡𝗧𝗘𝗟𝗟𝗜𝗚𝗘𝗡𝗖𝗘 𝟮.𝟬 ™ DOCUMENTATION.

🌐 NOTE: 𝗔𝗥𝗧𝗜𝗙𝗜𝗖𝗜𝗔𝗟 𝗜𝗡𝗧𝗘𝗟𝗟𝗜𝗚𝗘𝗡𝗖𝗘 𝟮.𝟬 ™ is part of MAGNETRON ™ TECHNOLOGY.

🌐 NOTE: REMEMBER 𝗔𝗥𝗧𝗜𝗙𝗜𝗖𝗜𝗔𝗟 𝗜𝗡𝗧𝗘𝗟𝗟𝗜𝗚𝗘𝗡𝗖𝗘 𝟮.𝟬 ™ ROBOTS WORK WELL TOGETHER (e.g HIVES, PHALANX, SWARM) AND CAN GATHER DATASETS (WITH EYE CAMERAS, EAR MICROPHONES ETC) FOR DEEP LEARNING (THIS MAKES THEM IDEAL FOR RECONNAISSANCE AS WELL AS ADAPTING TO SITUATION SPECIFIC THREATS IN MILITARY/LAW ENFORCEMENT/PERSONAL SECURITY ENDEAVORS etc).

🌐 NOTE: REMEMBER 𝗔𝗥𝗧𝗜𝗙𝗜𝗖𝗜𝗔𝗟 𝗜𝗡𝗧𝗘𝗟𝗟𝗜𝗚𝗘𝗡𝗖𝗘 𝟮.𝟬 ™ ROBOTS WORK WELL TOGETHER (e.g HIVES, PHALANX, SWARM) MAKING GATHERING IMAGES FOR ADVANCED ROBOTIC NeRF VIEW SYNTHESIS EASY.

🌐 NOTE: YOU CAN IMPLEMENT HALO FORGE MODE STYLE CONTROL FOR HUMANS TO INTERFACE WITH THIS PROXIA e.g FOR SECURITY SURVEYING. THIS IS AN ADVANCED FORM OF SECURITY/CYBERSECURITY.

🌐 NOTE: THIS PROXIA CAN BE USED TO CONFIRM HOW SOMETHING SHOULD/MIGHT LOOK IN A GIVEN LIGHTING SITUATION.

GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (NeurIPS 2022)
_{Official PyTorch implementation}

For business inquiries, please visit our website and submit the form: NVIDIA Research Licensing

News

• 2022-10-13: Pretrained model on Shapenet released! Check more details here
• 2022-09-29: Code released!
• 2022-09-22: Code will be uploaded next week!

Requirements

• We recommend Linux for performance and compatibility reasons.
• 1 – 8 high-end NVIDIA GPUs. We have done all testing and development using V100 or A100 GPUs.
• 64-bit Python 3.8 and PyTorch 1.9.0. See https://pytorch.org for PyTorch install instructions.
• CUDA toolkit 11.1 or later. (Why is a separate CUDA toolkit installation required? We use the custom CUDA extensions from the StyleGAN3 repo. Please see Troubleshooting) .
• We also recommend to install Nvdiffrast following instructions from official repo, and install Kaolin.
• We provide a script to install packages.

Server usage through Docker

• Build Docker image

cd docker
chmod +x make_image.sh
./make_image.sh get3d:v1

• Start an interactive docker container: docker run --gpus device=all -it --rm -v YOUR_LOCAL_FOLDER:MOUNT_FOLDER -it get3d:v1 bash

Preparing datasets

GET3D is trained on synthetic dataset. We provide rendering scripts for Shapenet. Please refer to readme to download shapenet dataset and render it.

Train the model

Clone the gitlab code and necessary files:

cd YOUR_CODE_PATH
git clone [e-mail]:nv-tlabs/GET3D.git
cd GET3D; mkdir cache; cd cache
wget https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan3/versions/1/files/metrics/inception-2015-12-05.pkl

Train the model

cd YOUR_CODE_PATH 
export PYTHONPATH=$PWD:$PYTHONPATH
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

• Train on the unified generator on cars, motorbikes or chair (Improved generator in Appendix):

python train_3d.py --outdir=PATH_TO_LOG --data=PATH_TO_RENDER_IMG --camera_path PATH_TO_RENDER_CAMERA --gpus=8 --batch=32 --gamma=40 --data_camera_mode shapenet_car  --dmtet_scale 1.0  --use_shapenet_split 1  --one_3d_generator 1  --fp32 0
python train_3d.py --outdir=PATH_TO_LOG --data=PATH_TO_RENDER_IMG --camera_path PATH_TO_RENDER_CAMERA --gpus=8 --batch=32 --gamma=80 --data_camera_mode shapenet_motorbike  --dmtet_scale 1.0  --use_shapenet_split 1  --one_3d_generator 1  --fp32 0
python train_3d.py --outdir=PATH_TO_LOG --data=PATH_TO_RENDER_IMG --camera_path PATH_TO_RENDER_CAMERA --gpus=8 --batch=32 --gamma=400 --data_camera_mode shapenet_chair  --dmtet_scale 0.8  --use_shapenet_split 1  --one_3d_generator 1  --fp32 0

• If want to train on separate generators (main Figure in the paper):

python train_3d.py --outdir=PATH_TO_LOG --data=PATH_TO_RENDER_IMG --camera_path PATH_TO_RENDER_CAMERA --gpus=8 --batch=32 --gamma=40 --data_camera_mode shapenet_car  --dmtet_scale 1.0  --use_shapenet_split 1  --one_3d_generator 0
python train_3d.py --outdir=PATH_TO_LOG --data=PATH_TO_RENDER_IMG --camera_path PATH_TO_RENDER_CAMERA --gpus=8 --batch=32 --gamma=80 --data_camera_mode shapenet_motorbike  --dmtet_scale 1.0  --use_shapenet_split 1  --one_3d_generator 0
python train_3d.py --outdir=PATH_TO_LOG --data=PATH_TO_RENDER_IMG --camera_path PATH_TO_RENDER_CAMERA --gpus=8 --batch=32 --gamma=3200 --data_camera_mode shapenet_chair  --dmtet_scale 0.8  --use_shapenet_split 1  --one_3d_generator 0

If want to debug the model first, reduce the number of gpus to 1 and batch size to 4 via:

--gpus=1 --batch=4

Inference

Inference on a pretrained model for visualization

• Download pretrained model from here.
• Inference could operate on a single GPU with 16 GB memory.

python train_3d.py --outdir=save_inference_results/shapenet_car  --gpus=1 --batch=4 --gamma=40 --data_camera_mode shapenet_car  --dmtet_scale 1.0  --use_shapenet_split 1  --one_3d_generator 1  --fp32 0 --inference_vis 1 --resume_pretrain MODEL_PATH
python train_3d.py --outdir=save_inference_results/shapenet_chair  --gpus=1 --batch=4 --gamma=40 --data_camera_mode shapenet_chair  --dmtet_scale 0.8  --use_shapenet_split 1  --one_3d_generator 1  --fp32 0 --inference_vis 1 --resume_pretrain MODEL_PATH
python train_3d.py --outdir=save_inference_results/shapenet_motorbike  --gpus=1 --batch=4 --gamma=40 --data_camera_mode shapenet_motorbike  --dmtet_scale 1.0  --use_shapenet_split 1  --one_3d_generator 1  --fp32 0 --inference_vis 1 --resume_pretrain MODEL_PATH

• To generate mesh with textures, add one option to the inference command: --inference_to_generate_textured_mesh 1

• To generate the results with latent code interpolation, add one option to the inference command: --inference_save_interpolation 1

Evaluation metrics

Compute FID

• To evaluate the model with FID metric, add one option to the inference command: --inference_compute_fid 1

Compute COV & MMD scores for LFD & CD

• First generate 3D objects for evaluation, add one option to the inference command: --inference_generate_geo 1
• Following README to compute metrics.

License

Copyright © 2022, NVIDIA Corporation & affiliates. All rights reserved.

This work is made available under the Nvidia Source Code License .

Broader Information

GET3D builds upon several previous works:

• Learning Deformable Tetrahedral Meshes for 3D Reconstruction (NeurIPS 2020)
• Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis (NeurIPS 2021)
• Extracting Triangular 3D Models, Materials, and Lighting From Images (CVPR 2022)
• EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks (CVPR 2022)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer (NeurIPS 2021)
• Nvdiffrast – Modular Primitives for High-Performance Differentiable Rendering (SIGRAPH Asia 2020)

Citation

@inproceedings{gao2022get3d,
title={GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images},
author={Jun Gao and Tianchang Shen and Zian Wang and Wenzheng Chen and Kangxue Yin
and Daiqing Li and Or Litany and Zan Gojcic and Sanja Fidler},
booktitle={Advances In Neural Information Processing Systems},
year={2022}
}