Project Title: Using markerless motion capture to drive music generation and develop neuroscientific/psychological theories of music creativity and music-movement-dance interactions
This is a new exploratory project that aims to use the recent advances in real-time markerless motion/pose capture software (MediaPipe, Metalab Livepose based on OpenPose and PoseNet, and potentially Jarvis/DeepLabCut) to create a working framework whereby movements can be translated into sound with short-latency, allowing for example, gesture-driven new musical instruments and dancers who control their own music while dancing. The project will require familiarity with Python, and ability to interface with external packages like MediaPipe and Jarvis. Familiarity with low-latency sound generation, image processing, and audiovisual displays is an advantage, though not necessary. The development of such tools will facilitate both artistic creation, as well as scientific exploration of multiple areas, including for example - how people engage interactively with vision, sound, and movement and combine their respective latent creative spaces. Such a tool will also have therapeutic/rehabilitative applications in populations of people with limited ability to generate music and in whom agency and creativity in producing music have been shown to produce beneficial effects.
Planned effort: 350 hours Skill level: Intermediate/Advanced Pre-requisite skills: Comfortable with Python. Experience with image/video processing and using deep-learning based image-processing models. Familiarity with C/C++ programming, low-latency sound generation, image processing, and audiovisual displays, as well as MediaPipe or other markerless pose/motion capture tools is an advantage, though not necessary. Tech keywords: Sound/music generation, Image processing, Pose estimation, Python, MediaPipe, Wekinator. Mentor: Suresh Krishna (more may be added later) No planned longer absences.