Movie Vis is a analytical approach to study hidden biases in film through the application of Power/Agency scoring, Bechdel Scoring and Character Network Mapping. Movie Vis studies these scoring methods through a gender and race lens.
Research on racial and gender bias in literature is impeded by limitations in acquiring traditional human ratings. Our project provides an analytic and visual aid to overcome these limitations and aid interdisciplinary research. Drawing from a collection of previously validated text analytics, and using large databases of film data, we will create an application that displays various metrics of racial and gender bias as they appear within the spoken lines of narrative characters within films. Recently, big data and NLP approaches have shown promise in aiding research on this topic, specifically by scraping films and book narratives from online sources.
The Movie Vis project is separated into four main components: data collecting, data processing, data analysis, and data visualization.
-
Data Processing: Data is collected through several sources. First, we scrape the movie script from The Internet Movie Script Database (IMSDb). Next we search for this script's metadata through the The Movie Database (TMDb) API. From TMDb we collect information such as the celebrities casted in the movie, the profile picture of each celebrity, the character that each celebrity played, the celebrities' gender, the year the movie was released, box office revenue, and so on. There are several celebrities in the TMDb database that are missing gender information, and the TMDb database does not collect race or ethnicity. To collect this we use modified version of the deepface framework to predict the gender and race of the celebrity. After this step we then map the celebrity metadata to the characters parsed from the script pulled via IMSDb. Through this we are able to assume the gender and race of the characters played in a film as well as other crucial information. After we've collected the film metadata we begin to analyze and parse the film script itself. We extract important identifiers such as the characters, dialogue, and scene description. We then map the the characters found in the script to the celeb metadata collected via TMDb through a fuzzy match approach using pylcs (or py_common_subseq if the user is on MacOS). Of course, this is not a perfect approach. For example, if the script being parsed is messy (as many scripts from IMSDb are) then there will be characters who are incorrectly mapped, or dialogue that is not recognized. Generally film writers follow a specific format, where character names follow the same level of indentation throughout the entire script. This also applies for dialogue, and other aspects of the script. Our script parsing and data collection relies on this. We've made every effort to circumvent issues were the script does not follow proper indentation. This can happen if the film script was originally a PDF that was converted to txt and uploaded to the IMSDb.
-
Data Processing: From the step above we were able to collect data on over 1200 films. Film scripts in this set that were poorly mapped to celebrity metadata were identified as a bad match. However, we included these films in our visualization to show the pitfalls of text analysis. The data collected were stored as json objects with TSV counterparts. These TSV files were then inserted into a postgres RDS. Relationships were built based on the TMDb_Id as well as the Celeb_Id, where the Celeb_Id is unique to a celebrity within a film. This diagram provides an idea of how to structure your dataset if using this project for your own analysis.
-
Data Analysis: Our analysis were separated into three parts: Bechdel Scoring, Power/Agency Scoring, and Character Graph Mapping.
- Bechdel Scoring If two or more women in a film are conversing, and the conversation is not about a male then this film generally passes the Bechdel test.
- Power/Agency Power & Agency is calculated based on the lines spoken in the film. To determine this we employed the NLTK library Natural Language Toolkit (NLTK).
- Character Network Mapping Characters are mapped in an undirected graph based on their appearance in certain scenes. As would be expected, weight is calculated as the number of scene occurrences for the source node.
-
Visualization We visualize the data and analysis collected through Tableau. We use Tableau public to publish our data and a frontend Angular.js app to compile our publications on to one site: movievis.com
This project has been designed to allow end users to easily collect film data for their own analysis. We've provided the source files of our analysis here. Running this project will collect data so that the user can implement their own analysis. Feel free to use our text analysis & network mapping as a part of your project, just remember to install the py packages for those scripts.
To use this project, download this repo to a local environment. We recommend using anaconda for this, but this will probably work in a raw python environment (just skip the anaconda specific steps). Navigate to the directory where you cloned the repository, then run the following:
conda create --name <environment> python=3.6
conda activate <environment>
pip install -r src/python/<platform>-requirements.txt
python -m spacy download en_core_web_sm
Please note, if you are using a virtual environment for this project you will need to install opencv-python in your local environment and copy this package from it's site package location over to the virtualenv.
Also note, this project installs tensorflow and its dependencies.
First, register for a developer account with the TMDb. Save this key as we will use it in our API calls.
To use Movie Vis run the wrapper py file. This script will start by scraping IMSDB for the movie script, and then pulling data from TMDB and parse the movie script for various elements, like characters and dialog.
E.g.
python src/python/process_scripts/wrapper.py --title <film_title> -o <output_directory> --tmdb_key <tmdb_key>
Example:
python src/python/process_scripts/wrapper.py --title "A Few Good Men" -o data --tmdb_key 1234567891010987654321
For more information use the --help argument to see a list of available options.
python src/python/process_scripts/wrapper.py --help
The data collected will be written to the folder you used in your -OUTPUT argument.
For the actual analysis data, the two relevant scripts are text_analysis.py and scene_analysis.py located in src/python/analysis/models. These rely on the output of wrapper.py above.
Examples:
python src/python/analysis/models/text_analysis.py -i data/dialog -o ./data/text_analysis
python src/python/analysis/models/scene_analysis.py -i data/dialog -o data/scene_analysis --cast_data data/film_characters.csv
We have created charts and visuals around the data collected for this project. The visualizations can be seen here: http://movievis.com