Pocket6 is a smartphone application using ARKit to turn regular iPhones into spatial input controllers for distant displays.
-
6DOF hand motion + touch inputs: Users can interact by performing 3D hand movements in 6DOF (translation + rotation) and can reliably segment and further enhance their gestures by touchscreen inputs.
-
Subtle gestures: Users can perform subtle hand motions within a small control space (e.g. 16 × 9 × 16 cm). This makes the gestural interaction more comfortable since users do not need to perform large hand motions.
-
Unconstrained user mobility: Users can freely and fluently change their position, orientation, or posture while using the system. An auto-calibration algorithm will adapt the control space so that the user will stay correctly mapped to the distant display application (no user-initiated calibration required).
-
No specialized hardware required: No external hardware, as external trackers or cameras, are required.
Interior Designer demo, included in the project.
Another demo, currently not included in the project.
Automated control space remapping.
Distant display app
- Clone or download the repository.
- Start Unity and open the Pocket6 Distant Display App project.
- Open the MainScene from the Scenes folder.
- Press Play.
- Click Start server in the app and let it run.
- Proceed with the Pocket6 smartphone app.
Smartphone app
- Open the Pocket6 Smartphone App project in another Unity instance.
- Select iOS as the target platform under File > Build Settings > Platform > iOS > click Switch Platform.
- Open the MainScene from the Scenes folder.
- Change the resolution of the Game window by selecting Phone X/XS 2436x1125 Portrait.
- Build the iOS app under File > Build Settings > Build > Create folder > Choose a name > click Save.
- In the created folder select Unity-iPhone.xcodeproj and open it in Xcode.
- Connect your iPhone to your Mac and select it as the build Device.
- In Xcode project settings under Signing & Capabilities select your Apple Developer Signing Team ID, Bundle Identifier, and press Play.
- Open the Pocket6 app on your iPhone.
- Enter the network address (IP) of the distant display Mac/PC. You can find it in the UI of the distant display app. Make sure, that both devices are in the same local network.
- Click Connect.
- Interact by moving your device-holding hand and observe the cursor on the distant display. Point at a furniture piece and press with your finger on the touchscreen to grab it. Alternatively, you can swipe your finger left or right to rotate the furniture piece.
- Use Change view on the smartphone, as well as the distant display, to see under the hood of Pocket6.
Advanced users, feel free to test various control space sizes and aspect ratios. These settings can be adjusted in the Pocket6 Logic game object in the smartphone app project.
The project was made and last tested in Unity 2021.3+, Xcode 14.3, iPhone 13 Pro and iOS 16.4.
System requirements:
- iPhone with ARKit capabilities
- Unity
- Xcode
- Mac or PC
- We use SAFullBodyIK to manipulate the humanoid rig.
- The deprecated Unity HLAPI networking was changed to Mirror.
- Improve network lag by using SmoothSync.
- Improve input precision by using the 1€ filter.
- Save the network addresses using the device's internal storage or Easy Save.
This work is based on a publication "Pocket6: A 6DoF Controller Based On A Simple Smartphone Application" in the ACM SUI '18: Proceedings of the Symposium on Spatial User Interaction 2018. The publication also includes related work, user studies, applications, and future work directions.
Copyright (C) 2023 Teo Babic