(submitted as a Poster presentation to International Symposium on Computational Media Art (ISCMA) 2019)
Contributions for this project:
- 3D model: Martina Jole Morro
- Engineering: Suzana Ilic
- Video editing: Johannes Felder
- Music: Christian Anich
- Overvoice: Josiah Sampson
Artificial Intelligence, Virtual and Augmented Reality are transforming multimedia art, offering the opportunity for novel creative human-machine collaborations.
This is the synthesis of a character-level long short-term memory network for generating poetry and L.eO. (Luci e Ombre; Lights and Shadows), a virtual landscape composed of dynamic architectural elements and surfaces.
Recurrent neural networks (RNNs) encompass high-dimensional and non-linear hidden states and are able to iterate over sequences of arbitrary size, process and memorize information. Among other models, RNN variants are commonly implemented in the field of natural language generation. As a widespread extension, we trained a character-level long short-term memory network (LSTM) on a dataset of 1.3M characters of classical and contemporary poems.
Deep learning architecture:
- Single LSTM (128 hidden units)
- Dropout layer (0.2)
- Dense layer; softmax activation function
- Optimizer: Adam; learning rate: 0.0005; loss: categorical cross-entropy
The selected poem was generated during epoch 105 and shows errors in morphology and syntax, but seems largely coherent from a semantic perspective, where expressions like soul of the storm can be interpreted as metaphors. The linguistic style matches approximately the structure and aesthetics of poetry, however, there is a striking word-level repetitiveness. The poem was narrated by a human and recorded to be merged with the video sequence.
Poem generated during epoch 105:
on a charred spinning wheel, the world was cold the soul of the storm,
the shadow s soul where the strong she still,
the stars that beautiful and strain,
and the strange and the storm of the stars,
and the stars of the storms of the stars,
i say i shall be the made the stars of the storm,
the stars when the wind of the stream of the shadow,
the thing of the said the world was a sea,
and the shadow of the sky
L.e.O is an alternate reality composed of light and shadows. 9 selected silhouettes were casted and extruded onto the original structure. The results were then transformed into architectural and landscape objects. The objects were deconstructed, analyzed an reassembled in a different manner. Subsequently, the pieces were scaled up and down depending on their role in the environment. The entire digital island is made of the same selection of the original 9 shadows using a modular system.
The digital island was developed in Rhinoceros 3D, a design application software, and then imported into Unity for adding a range of different textures to the environment.
There are five main textures in our virtual reality:
- Standard texture, some with emissive light, others with an additional coloured spotlight
- Grandaunt texture, between two colours
- Moving texture, where the texture takes over the mash and moves it resembling water
- Glitched texture, where the redeeming hierarchy was compromised creating compelling visual effects in the final result
- Grid textures
As a final step, we blended the audio recording of the AI poem into the video sequence, which gives the illusion of exploring a virtual environment that is completely different from our own reality, while hearing the artificially generated poem.
Creative design and visualization projects can be enhanced by Artificial Intelligence in various ways, such as leveraging deep learning models for image, video, text and speech generation. Thus, it can be used for content creation as well as for assisting the human in the creative process. This project demonstrates how two creative streams can be merged: (1) a 3D model of a digital island of nine original shapes, created through modular and patchwork assembly, that seem to defy the laws of physics, and (2) a poem generated by a character-level LSTM trained on a dataset of 1.3M characters of poems. Future work can include deep learning models such as Generative Adversarial Networks for generating novel virtual landscapes.
Code based on Keras implementation
-
Sutskever, I., Martens, J. and Hinton, G.E.: Generating text with recurrent neural networks (Proceedings of the 28th International Conference on Machine Learning ICML-11, 2011), 1017-1024.
-
Gatt, A., and Krahmer, E.: Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation. (Journal of Artificial Intelligence Research 61, 2018), 65-170.
-
Ghang, L., Eastman, C.M., Taunk, T. and Ho, C.: Usability principles and best practices for the user interface design of complex 3D architectural design and engineering tools (International Journal of Human-Computer Studies 68, no. 1-2, 2010) 90-104.