This project is a study on Aesop Fables and the possibility of exploring the connections between them in a new and fun way using machine learning.
Demo at letsreadastory.itayniv.com
This project is a study in which I chose to focus and analyze Aesop Fables to produce new and interesting juxtapositions between sentences to create new stories. I was drawn to Aesop Fables texts because of their concise yet rich story lines, the use of animals as metaphors, and the strong morals embedded in each story.
Each original Aesop Fable contains:
- A short title, usually very descriptive of the story's content and characters.
- The story itself, usually no more than 30 sentences.
- The moral of the story, usually contains a metaphor built on the inherent nature or trait of the animals in the story.
For the analysis of the content I compiled a JSON file holding all stories broken down to individual sentences, their titles, characters, and animals.
This file is key for generation of new stories, as it holds all sentences and acts as the 'database' for the experiment.
Using Google's Universal Sentence Encoder, I analyzed all sentences derived from the fables(~1500 sentences).
This yields a JSON file containing sentence embeddings for each sentence in a 512 dimensional space, this is the similarity map I use to compare and generate new juxtapositions.
Example line from the file:
{"message": "There was once a little Kid whose growing horns made him think he was a grown-up Billy Goat and able to take care of himself.", "message_embedding": [0.06475523114204407, -0.026618603616952896, -0.05429006740450859, 0.003563014790415764 ...........,0.06475523194004407]}
For processing and retrieval of information for similarities, averages and distances between sentences I used the ML5 Word2Vec class and changed it a bit to work with the universal sentence encoder scheme.
To enrich the stories, I'm using Google Magenta's sketch-RNN model: A Generative Model for Vector Drawingsto reconstruct illustrations from a pre trained model to accompany the generated stories.
Using simple RegEx search, the javascript functionality determines which animal appears in the generated story and then draws the illustration from the trained sketch RNN model using P5. If the sentence contains an animal that does not exist in the model, there is another function that 'enriches' the model's keywords and matches similar animals specified in the sentence.
These illustrations then become musical phrases based on some pre determined rules:
-
With the help of AFINN-based sentiment analysis library I analyze each sentence and determine whether it has a positive or negative sentiment. Based on that sentiment (a score between -5 to 5), I'm mapping the illustration's X and Y coodinates to musical notes on a major or minor B scale - positive scores get a major scale 😊 and negative scores get a minor scale 😥.
-
According to the animal appearing in the sentence, I choose a different tone.js synthesizer and a different number of musical notes. For example, an predatory animal that tends to be scary, like a wolf, will be played by a low-tone synth and a small amount of musical notes sounds. Conversely, a bird will be played by a high-pitched synth and a higher amount of sounds.
This method, of course, does not purport to represent the story reliably, and there are cases in which there will be no match between the musical sounds and the story, but it gives a certain touch to the story and enriches the characters and illustrations in a somewhat charming way. In future versions this method will need to be improved.
This is an ongoing projects I intend to work on in the next couple of months, hopefully it will become my thesis project in ITP-NYU. I intend to develop more tools to help structure stronger narratives with the help of these and other machine learning algorithms. Testing and deployment on other platform that give better focus to different aspects of the storytelling practice – actual books, but also smart devices, speakers, tablets and other immersive media devices.