Geronymos / AR-Desktop

A web app that shows windows and websites in the real world.

AR-Desktop 💻

Mixed reality for mobile working

About ❕

In this work I would like to create an application that enforces augmented reality in the broad masses, which is not only for fun, but also for work and does not require large investments. This should be possible with a device that everyone already has: the smartphone.

For this purpose, I am developing a mixed reality application in a stereoscopic view as a browser extension for smartphones, with which the limits of the monitor can be left and the dimensions of the room can be used to work independently of location.

You interact with the usual applications and websites in the form of virtual objects that merge with the real world via the direction of your gaze, gestures or external devices.

Goals 🏁

• mobile 🏃
• cheap/free 💸
• productive 📊
• FUN 🥳

Interactions 💪

• Key inputs ⌨🕹
• Movement inputs 📱
• Tracking 📸
• Brain interface 🧠

Getting Started

Installing 📲

1. Download or clone this repo with

git clone https://github.com/Geronymos/AR-Desktop.git

2. Build the browser extension.

⚠ You need to have node.js and npm installed!

npm install
npm run build-ext

3. Now install the extension
   • Firefox 🦊: about:addons > ⚙ > Install Add-on From File
   • Chromium 🧿: chrome://extensions/ > 🎚 Developer mode > load unpacked

Usage

1. Generate barcode markers 🔳 for each tab (0 to ...) and print them 🖨 or show them on a screen 💻.

2. Open your favorite web application and see it in the real world 🏞 by clicking on the browser action next to the url.

3. The website will appear as a head-up display and will show in place of the barcode marker 🔳.

Develop 🛠

# For chromium
npm run chrome
# For Firefox
npm run firefox

Galery 🖼

⚒ Build with

• A-FRAME [🐙] - 🅰 web framework for building virtual reality experiences.
  • Three.js [🐙] - JavaScript 3D library.
• AR.js [🐙] - Image tracking, Location Based AR, Marker tracking. All on the Web.
• web-ext [🐙] - A command line tool to help build, run, and test web extensions
• Webpack [🐙] - A bundler for javascript and friends.

Roadmap

• tab video streaming
• click event transmittion
• Keyboard input
• multiple tabs
• use environment camera in multi camera setups
• stereoscopic camera
• virtual keyboard
• Firefox/Mobile support
• Hand-tracking
• 3D-Object-Repo

Future 👾

• Tensorflow Handpose - Pretrained models for TensorFlow.js
  • Fingerpose 🐙 - Finger pose classifier for hand landmarks detected by TensorFlow.js handpose model
• Mediapipe [❔] [npm] - MediaPipe offers cross-platform, customizable ML solutions for live and streaming media.
• Superhands 🐙 - 👐All-in-one natural hand controller, pointer, and gaze interaction library for A-Frame

Related Projects 🔗

• https://www.npmjs.com/package/web2vr
• https://www.npmjs.com/package/aframe-arjs-vr
• https://www.npmjs.com/package/aframe-html
• https://www.npmjs.com/package/aframe-html-shader
• https://github.com/mozilla/webextension-polyfill#installation

Authors

• Gero Beckmann - Initial work - Geronymos

License

This project is licensed under the GPT-3 License - see the LICENSE file for details

Acknowledgements

Slack Mediapipe

GitHub, Stackoverflow, YouTube, Medium, Reddit

Presented at