jordanbCS / NatureVR

Making the natural treasures of our world accessible to everyone

Nature VR

Making the natural treasures of our world accessible to everyone. The project will take 2d images of natural scenes, and attempt to texture 3d Unity environments, using a complex of machine learning approaches for interpolating texture and depth from static images.

Motivation

• https://psmag.com/environment/living-near-a-forest-is-linked-to-better-brain-health?source=TDB&via=FB_Page

Terrain generation models

• https://github.com/universome/alis

2d --> 3d models

• Google PHORHUM - "...Observing that 3D supervision alone is not sufficient for high fidelity color reconstruction, we introduce patch-based rendering losses that enable reliable color reconstruction on visible parts of the human, and detailed and plausible color estimation for the non-visible parts..." - https://phorhum.github.io/

• NVIDIA's New AI: Enhance! - https://www.youtube.com/watch?v=e0yEOw6Zews

• "AI can turn a collection of 2D images into an explorable 3D world" - https://www.newscientist.com/article/2294737-ai-can-turn-a-collection-of-2d-images-into-an-explorable-3d-world/ - https://www.youtube.com/watch?v=BBmcsyB7aDw

• https://www.reddit.com/r/MachineLearning/comments/ruaew1/r_learning_3d_representations_from_2d_images/

• https://www.reddit.com/r/MachineLearning/comments/pskwhy/r_3d_annotation_of_arbitrary_objects_in_the_wild/

• "AI Generates 3D high-resolution reconstructions of people from 2D images | Introduction to PIFuHD" https://www.youtube.com/watch?v=ajWtdm05-6g

• "PIFuHD: High-Resolution 3D Human Digitization (CVPR2020 Oral, 5min Presentation)"
  https://www.youtube.com/watch?v=uEDqCxvF5yc

• https://github.com/facebookresearch/pifuhd

• https://colab.research.google.com/drive/1cPntk-PyiXnxiUfsfw5teV8UrdPoi0e4?usp=sharing

Existing Nature VR applications

• "Get Lost In Nature With Luke" https://www.oculus.com/experiences/rift/3507999942601704/ - https://www.reddit.com/r/virtualreality/comments/nubeqv/get_lost_in_nature_with_luke_how_is_this_level_of/ - https://www.youtube.com/watch?v=j4RX_TnJH0k
• "Nature Treks VR" - https://www.oculus.com/experiences/quest/2616537008386430

Related

• https://www.reddit.com/r/InternetIsBeautiful/comments/nuc3ef/this_website_developed_by_nvidia_lets_you_draw/h0xbuhp/

• https://github.com/cyrildiagne/instagram-3d-photo

• https://www.reddit.com/r/MediaSynthesis/comments/fq6zyx/front2back_ai_reconstructs_3d_shapes_from_2d/

• https://www.reddit.com/r/MachineLearning/comments/fc9l4p/n_turn_2d_photos_into_3d_using_convolutional/

• https://github.com/NVIDIAGameWorks/kaolin/

• https://smaerdlatigid.github.io/3D-Photo-Viewer/

• https://www.reddit.com/r/replications/comments/e17lr4/turning_2d_into_3d_with_ai_depth_map/

• https://arxiv.org/abs/1811.07605 sparse 3d to 3d point cloud

• Eric Guérin, Julie Digne, Eric Galin, Adrien Peytavie, Christian Wolf, et al.. Interactive Example- Based Terrain Authoring with Conditional Generative Adversarial Networks. Transactions on Graph- ics (Proceedings of Siggraph Asia 2017), ACM, 2017, 36, pp.228 - 228. <10.1145/3130800.3130804>.
  

• BBC Earth - Live in VR - https://www.youtube.com/watch?v=k9OkENZwzL4

• https://www.reddit.com/r/virtualreality/comments/a340lk/research_at_nvidia_the_first_interactive_ai/

• https://github.com/SyntonicApps/unity-vrtk-examples - https://www.youtube.com/watch?v=aJhPm5h1pl4

• https://github.com/llSourcell/How-to-Generate-Art-Demo

  • https://medium.com/towards-data-science/neural-networks-and-the-future-of-3d-procedural-content-generation-a2132487d44a - "Knowing that, I went over to http://terrain.party, a website where you can download elevation data from the Earth."

• https://github.com/jiajunwu/3dinn - "Wu, J., Xue, T., Lim, J. J., Tian, Y., Tenenbaum, J. B., Torralba, A., & Freeman, W. T. (2016). Single image 3D interpreter network. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 9910 LNCS). http://doi.org/10.1007/978-3-319-46466-4_22"

• https://www.reddit.com/r/virtualreality/comments/65ctm9/vr_biking_in_the_mountains_with_vr_rider_early/

• http://variety.com/2017/digital/news/adobe-6dof-vr-video-algorithms-1202394491/

  • http://zhilichen.com/projects/sixdofvr/vr360.pdf

• https://research.googleblog.com/2017/07/using-deep-learning-to-create.html