Final project for 6.835: Intelligent Multimodal User Interfaces
- multimodal interactions using gestures & speech for d3 visualizations
- Install Node&Npm here
- Install Leapmotion daemon here
- In one terminal:
$ sudo leapd
- In a different terminal:
$ npm install --save $ npm run server
This will run webpack, build your files in the dist/ folder, and start a python simple server. You can access the following visualizations by visiting the corresponding links in your browser:
Force-directed graph visualization: http://localhost:8080/force.html
interact with this by
- moving nodes with grab gesture
- if cursor intersects node can change color of node by saying "color red", "color green", "color blue"
- if cursor intersects node can remove node by saying "delete node"
- can reset colors of nodes by saying "reset nodes"
Scatter Graph pan+zoom visualization: http://localhost:8080/scatterPan.html
interact with this by
- panning by making grab gesture and moving fist
- zooming in by holding hand at a location on screen and saying 'increase zoom'
- zooming out by holding hand at a location on screen and saying 'zoom out'
Scatter Graph gesture zoom visualization: http://localhost:8080/scatterZoom.html
interact with this by
- zooming in by pointing at a location on screen and doing the Leap Motion built-in screenTap gesture
- move finger forward and back as if ringing a doorbell
- zooming out by saying 'zoom out'
Put-that-there visualization: http://localhost:8080/editor.html
interact with this by
- pointing to location on screen and say
- "create" or "put"
- a color "purple", "red", "blue" are the ones supported
- "here"
- e.g. - "put purple circle here"
MouseEvent emulator visualization: http://localhost:8080/click.html
interact with this by
- pointing at a location on screen and doing the Leap Motion built-in screenTap gesture
- if recognized this will show by triggering a MouseEvent that clicks on the top left circle and switches its color
| File or Folder Name | Purpose |
|---|---|
| package.json | lists project settings and dependencies |
| webpack.config.js | has build settings for webpack, input files are in src/ and output files in dist/ |
| README.md | documentation |
| scripts/start-server.sh | used in npm run server, builds webpack and starts python simple server |
| src/click-sim.js | javascript file for click.html visualization |
| src/force.js | javascript file for force.html visualization |
| src/put-that-there.js | javascript file for editor.html visualization |
| src/scatter-pan.js | javascript file for scatterPan.html |
| src/scatter-zoom.js | javascript file for scatterZoom.html |
| src/zoom.css | css file used in scatterPan.html and scatterZoom.html |
| dist/ | Webpack built js files, that are used in the properly named .html files |