BirukTes / The-Colour-of-Music

The Colour of Music - Unity 2D Visualisation demonstration application

The Colour of Music

The Colour of Music - Unity 2D Visualisation demonstration application

This project is about mapping or translating music into a visual experience such as a 2D colour picture, coloured video or light visualisation using Emotion as the medium (between music and colour). The system is a Windows desktop application built using Unity 3D Game Engine with C# and Sonic Annotator console application with Vamp Plugins for extracting audio data. (Take a below for more details.)

Application Features

• Detection the emotion of a piece of an audio
• Display a corresponding colour for a given emotion
• Playing the current audio
• An option to select between a 'Video' and 'Picture' mode
  • Moving colour change or picture.
  • A still picture, made up of all the screenshots taken during 'Video' mode.
• An option to change between selected audio file
• A save button to save the selected video (recorder is buggy, is unable to delete recorded video and audio after merging into the final output video) or picture
• A switch to full screen or windowed mode

Known Issues

• While switching to windowed mode from screen, a second window may appear in front which looks like not responsive:
  • Change to the first window from the taskbar or alt+tab switcher and close the second window.

System Requirements

1. Windows 10 64bit
2. 6 Gigabytes (GB) or more available RAM
3. Other CPU and GPU requirements:
   • https://docs.unity3d.com/2020.3/Documentation/Manual/system-requirements.html
   • 

Application Interface

• On startup

Output of Picture Mode

• Test audio files can be found in the releases folder (\The Colour of Music_Data\StreamingAssets\TCM\Audio).
• Each colour square is a screenshot from the 'video'/movie mode.

How it works?

1. Music is processed using Sonic Annotator to extract the Tempo and Mode, provided as JSON format/type.
2. Then as the music plays, the time position is taken, using that, the Tempo and Mode are looked-up and saved into currentTempo, currentMode... variables.
3. Using the current tempo and mode the emotion is determined.
4. Using the emotion the corresponding colour is shown using a LERP (Linear Interpolation) function to ease it.
5. There is also an intensity Synchronisation function which corresponds the current tempo, modifies the intensity of the colour shown.

Music and Emotion Mapping

• Emotion categories and methods are derived from the paper by (Ramos et al., 2011).

Emotion and Colour Mapping

• The emotion to colour map was created in accordance with (Art Therapy, 2011) and (Ram et al., 2020).

Results

1. After running the test cases, it was evident that the demonstrative system was able to detect emotion and display the corresponding colour successfully.
2. It was shown that some elements of Für Elise (Beethoven) were captured as 2D colour picture.

Limitations and Future Work

1. Emotion detection and use of colour palette were some of the limitations of this project and other previous projects due to ambiguities, subjectivities, and disagreements that exist (Yang & Chen, 2012).
2. Emotion detection could possibly use newer methods of MER which are mostly based of Machine Learning technology.
   • This would eliminate use of Sonic Annotator and conditionals and broaden the application of the system.
3. Colour changing could match the beat/tempo, currently it is not convincing that it does use the tempo multipler.
4. The prototype application could use less system resources, therefore being able to run and adapt to the length of the audio music played.
5. The video recording and saving feature could use the newer or another enhanced library.

References

• RAMOS, D., BUENO, J.L.O. & BIGAND, E., 2011. Manipulating Greek musical modes and tempo affects perceived musical emotion in musicians and nonmusicians. Brazilian Journal of Medical and Biological Research. 44(2), pp.165-172. Available from: https://doi.org/10.1590/S0100-879X2010007500148.

• RAM, V., SCHAPOSNIK, L.P., KONSTANTINOU, N., VOLKAN, E., PAPADATOU-PASTOU, M., MANAV, B., JONAUSKAITE, D. & MOHR, C., 2020. Extrapolating continuous color emotions through deep learning. Physical Review Research. 2(3), pp.033350. Available from: https://doi.org/10.1103/PhysRevResearch.2.033350.

• ART THERAPY. ,Color Psychology: The Psychological Effects of Colors, 2011.: Art Therapy. -02-28T06:49:08+00:00 [viewed Apr 4, 2021]. Available from: http://www.arttherapyblog.com/online/color-psychology-psychologica-effects-of-colors/.

• YANG, Y. & CHEN, H.H., 2012. Machine Recognition of Music Emotion. ACM Transactions on Intelligent Systems and Technology (TIST). 3(3), pp.1-30. Available from: https://doi.org/10.1145/2168752.2168754.

• PALMER, S.E., SCHLOSS, K.B., XU, Z. & PRADO-LEÓN, L.R., 2013. Music–color associations are mediated by emotion. Proceedings of the National Academy of Sciences of the United States of America. 110(22), pp.8836-8841. Available from: https://doi.org/10.1073/pnas.1212562110.

• WHITEFORD, K.L., SCHLOSS, K.B., HELWIG, N.E. & PALMER, S.E., 2018. Color, Music, and Emotion: Bach to the Blues. I-Perception (London). 9(6), pp.204166951880853-2041669518808535. Available from: https://doi.org/10.1177/2041669518808535.