This project is a framework for xR creators to extend inputs and outputs easily beyond the capabilities of conventional devices.
The framework consists of firmware for M5Stack and an SDK for Unity with the STYLY plugin. The M5Stack toolkit allows you to easily create IoT projects by connecting various sensors and actuators without soldering. The controller functions as a Bluetooth keyboard and a Wi-Fi server, sending keyboard events to a device such as a smartphone in response to input from sensors. Moreover, you can control actuators by sending HTTP requests to the controller.
With this framework, you may be able to expand your imagination of the concept of xR and realize your ideas as works.
本プロジェクトは、xRクリエーターが、従来のデバイスの能力を超える入出力を簡単に拡張するためのフレームワークです。
フレームワークは、M5Stack用のファームウェアと、STYLYプラグインを組み込んだUnity用のSDKで構成されています。M5Stackは、各種センサーやアクチュエーターをハンダづけなしで接続し、IoTプロジェクトを簡単に作成できるツールキットです。コントローラは、BluetoothキーボードおよびWi-Fiサーバーとして機能し、センサーからの入力に応じてキーボードイベントをスマートフォンなどのデバイスに送信します。さらに、コントローラにHTTPリクエストを送ることで、アクチュエータを制御できます。
このフレームワークを使うことにより、xRの概念に対する想像力を膨らませ、アイデアを作品として実現できるかもしれません。
🚧 Heavily work in progress; therefore, only for experimental purposes 🚧
- Mini Dual Button Unit
- Light Sensor Unit
- Mini Angle Unit
- I2C Joystick Unit V1.1
- Distance Ranging Sensor Unit
- TVOC/eCO2 Gas Sensor Unit
- RFID 2 Unit
- Gesture recognition sensor
- 12 Key Capacitive I2C Touch Sensor V3 by Seeed Studio
- iOS/iPadOS - STYLY app
- Android - STYLY app
- Oculus Quest 2 - STYLY app (from App Lab)
- Windows - STYLY VR app
- Windows - Chrome
- macOS - Chrome
The connected Unit(s) input is transmitted on three different channels: analog, joystick, and buttons. The analog and joystick channels can handle up to one sensor Unit input for each at once, and the buttons channel can handle up to six button inputs simultaneously.
Pattern \ Unit | Gesture | Joystick | Touch | RFID | Gas | Ranging | Dual Button | Analog In | Servo/Vibrator |
---|---|---|---|---|---|---|---|---|---|
A | 💡 | 💡 | 💡 | ||||||
B | 💡 | 💡 | 💡 | ||||||
C | 💡 | 💡 | |||||||
D | 💡 | 💡 | 💡 | 💡 | |||||
E | 💡 | 💡 | 💡 | 💡 | |||||
F | 💡 | 💡 | 💡 | 💡 | |||||
G | 💡 | 💡 | 💡 | 💡 | |||||
H | 💡 | 💡 | 💡 | 💡 | |||||
I | 💡 | 💡 | 💡 | 💡 | |||||
J | 💡 | 💡 | 💡 | ||||||
K | 💡 | 💡 | 💡 | 💡 | |||||
L | 💡 | 💡 | 💡 | 💡 | |||||
M | 💡 | 💡 | 💡 | ||||||
Port |
A |
A |
A |
A |
A |
A |
B |
B |
B |
- If you want to connect multiple Units to port A, please connect via a 1 to 3 HUB Expansion Unit.
- When a Ranging Sensor Unit is connected, you can't use a Joystick Unit nor Gas Sensor Unit.
- To change the connection, make sure to turn off the power (i.e., M5Stack Fire: double-clicking the power button, M5Stack Core 2: pressing the power button for 6 seconds). You cannot turn it off while your controller is connected to a USB port.
- Download M5Burner v3.0 for your platform from the official website
- Extract and launch the M5Burner (move to your Applications folder before launching on macOS)
- Choose "I/O Framework for xR" from the projects
- Click on the
Download
button of the project - Once finished downloading, the
Download
button becomes theBurn
button - Click on the
Burn
button - Choose the serial port in the
COM
field and click on theStart
button to burn the firmware
- Install EspTouch for iOS or Android (choose the
esptouch-v2.0.0.apk
) to your smartphone - Connect your smartphone to the Wi-Fi router
- Open your EspTouch app and tap on the EspTouch item (not EspTouch V2)
- Input the router’s password on the EspTouch app
- Power on (or reboot) your M5Stack and press the
A
button within three seconds1 - Tap the
Confirm
button on the EspTouch app and wait for a while - You will see an IP address on the screen of the M5Stack
- Choose a pattern from the table above and connect Unit(s) to your M5Stack controller (e.g., M5Stack FIRE)
- Power on your M5Stack controller
- If you want to use a Unit to be connected to Port B, please refer to the "How to setup" section and setup (Units to be connected to Port A will be recognized automatically)
- Connect the controller as a Bluetooth device to your device (please follow standard instruction for the device)
- Open the IOFrameworkWidget (BLE and Wi-Fi) scene in a browser and bring the browser frontmost
- Press the
Send
(C
) button to start sending - Control the joystick, sensor, etc.
- Once confirmed, press the
Stop
(C
) button again to stop sending
- Press the
Setup
(A
) button to enter the preferences screen - Press the
Next
(C
) button (if needed) to choose thePORT B: NONE
line - Press the
Go
(B
) button - Press the
-
(A
) or+
(C
) button to be matched to the device connected to the Port B (i.e.,DUAL BUTTON
,ANALOG IN
2,SERVO
orVIBRATOR
) - Press the
Done
(B
) button - Press the
Exit
(A
) button to back to the main screen
- Press the
Setup
(A
) button to enter the preferences screen - Press the
Next
(C
) button (if needed) to choose theRFID 1: **:**:**:**
line - Press the
Go
(B
) button - Press the
Reset
(C
) button - Put an RFID Tag on the RFID Unit, then remove the Tag
- Press the
Next
(C
) button to choose the next line - Repeat from step 3 for 3 times to register the remaining 3 RFID Tags
- Press the
Exit
(A
) button to back to the main screen
- "IO Framework M5" is shown as connected but no input to the STYLY scene. → Please try to unpair the controller in the Bluetooth preference and pair it again.
- Bluetooth connection status on my controller keeps switching between
Connected
andDisconnected
when not connected. → The controller might have been paired with an old host (i.e., a PC or smartphone). If you no longer use the controller with the host, please remove the device from the host.
The SDK consists of components as follows.
IOFrameworkManager
is a manager for the I/O FrameworkIOFrameworkTestUI
is a UI to test the frameworkIOFrameworkAnalogHandler
is a handler for events in the analog channelIOFrameworkJoystickHandler
is a handler for events in the joystick channelIOFrameworkButtonsHandler
is a handler for events in the buttons channelIOFrameworkGestureHandler
is a handler for events from a Gesture sensor (in both joystick and buttons channel)IOFrameworkOutputHandler
is a handler for events in the output channel via Wi-Fi
- Import the
IOFrameworkSDK.unitypackage
file to your Unity project - Instantiate an
IOFrameworkManager
- Instantiate an
IOFrameworkTestUI
as a Child of theIOFrameworkManager
object if necessary - Instantiate a handler from handlers in the SDK (e.g.,
IOFrameworkAnalogHandler
) as a Child of theIOFrameworkManager
object - Unpack the handler
- Navigate PlayMaker editor to the FSM of the handler and edit for your scene
How to layout the SDK components in Unity
IOFrameworkManager broadcasts events to all Children of the IOFrameworkManager.
ANALOG VALUE CHANGED
Int: analog valueJOYSTICK VALUE CHANGED
String: joystick valueBUTTON 1 DOWN
BUTTON 1 UP
BUTTON 2 DOWN
BUTTON 2 UP
BUTTON 3 DOWN
BUTTON 3 UP
BUTTON 4 DOWN
BUTTON 4 UP
BUTTON 5 DOWN
BUTTON 5 UP
BUTTON 6 DOWN
BUTTON 6 UP
Send events to IOFrameworkManager: Event Handlers FSM
.
SET OUTPUT VALUE REQUEST
String: output value
IOFrameworkManager broadcasts events to all FSMs.
HTTP RESPONSE
String: HTTP response
Send events to IOFrameworkManager: Event Handlers FSM
.
SET IP ADDRESS
String: IP address
Send events to IOFrameworkManager: HTTP Request Loop FSM
.
START LISTENING
STOP LISTENING
LISTEN ONCE
sequenceDiagram
participant S as STYLY app
participant M as M5Stack
M-->>S: keyboard events over BLE
The keys used for the protocol
- Keys in green: used in STYLY Web Player
- Keys in yellow: used for the Analog channel
- Keys in orange: used for the Joystick channel
- Keys in red: used for the Buttons channel
Analog value | Key |
---|---|
0 | ` |
1 | 1 |
2 | 2 |
3 | 3 |
4 | 4 |
5 | 5 |
6 | 6 |
7 | 7 |
8 | 8 |
9 | 9 |
10 | 0 |
Joystick | Key |
---|---|
Left-Up | y |
Up | u |
Right-Up | i |
Left | h |
Center | j |
Right | k |
Left-Down | n |
Down | m |
Right-Down | , |
No. | Name | Key | Assigned input |
---|---|---|---|
1 | Fire1 | v |
Dual Button - Red |
2 | Fire2 | b |
Dual Button - Blue |
3 | Fire3 | f |
Touch Sensor - CH0 |
4 | Jump | g |
Touch Sensor - CH1 |
5 | r |
Touch Sensor - CH2 | |
6 | t |
Touch Sensor - CH3 |
Joystick | Key |
---|---|
Up | u |
Left | h |
Right | k |
Down | m |
No. | Name | Key | Assigned input |
---|---|---|---|
1 | Fire1 | v |
Gesture - Forward |
2 | Fire2 | b |
Gesture - Backward |
3 | Fire3 | f |
Gesture - Clockwise |
4 | Jump | g |
Gesture - AntiClockwise |
5 | r |
Gesture - Wave |
sequenceDiagram
participant S as STYLY app
participant M as M5Stack
S-->>M: GET HTTP Requests over Wi-Fi
M-->>S: HTTP Responses over Wi-Fi
GET
http://{ip_address}/input
{analog value},{joystick value},{button 1 value},{button 2 value},{button 3 value},{button 4 value},{button 5 value},{button 6 value},
Example:
9,Left,0,1,0,0,0,0
GET
http://{ip_address}/output?val={value}
SERVO
:{value}
is servo angle in degree, between 0 and 180VIBRATOR
:{value}
is on duration in ms, between 0 and 100
Example:
GET
http://192.168.0.10/output?val=123
An example of controlling output without SDK in PlayMaker: build an URL (this part can be parameterized) and issue an HTTP request (you need to add states to handle errors properly)
- English: Arduino IDE environment - M5Core
- Chinese (simplified): Arduino IDE 环境搭建 - M5Core
- M5Unified v0.0.7 by M5Stack
- M5GFX v0.0.20 by M5Stack
- ESP32 BLE Keyboard library v0.3.0 by T-vK
- VL53L0X library for Arduino v1.3.1 by Pololu3
- Adafruit SGP30 Gas / Air Quality I2C sensor v2.0.0 by Adafruit
- Adafruit MPR121 Library v1.1.1 by Adafruit
- ESP32Servo v0.11.0 by Kevin Harrington and John K. Bennett
- ServoEasing v3.0.0 by Armin Joachimsmeyer
- DFRobot_PAJ7620 v1.0.1 by DFRobot
- Download the ZIP file at the ESP32-BLE-Keyboard v0.3.0 page (you don’t have to extract it after downloading)
- In the Arduino IDE, navigate to Sketch → Include Library → Add .ZIP Library...
- Select the file you just downloaded
- In the Arduino IDE, navigate to Tools → Manage Libraries...
- Type in a part of each library (e.g.,
VL53L0X
,Adafruit SGP30
etc.) in the text field in the top right corner, choose the right library and press the install button - Repeat the second step for all required libraries
- The included MFRC522 I2C Library is from the RFID_RC522 example in the public domain, originally developed by arozcan based on the findings of the pioneers and modified by M5Stack.
- The technique for detecting that an RFID tag has been removed was implemented by referring to the example proposed by uluzox and modified for cooperative multitasking
Footnotes
-
If you need to submit the MAC address to the administrator to connect to the network, please take a picture of the string on the startup screen below the version number and read it. ↩
-
If you want to connect Units to be connected to Port B and use
A.OUT
such as LIGHT, ANGLE etc., please choose this option. ↩ -
Make sure to install the one by Pololu, not by Adafruit. ↩