MovieQA

MovieQA: Understanding Stories in Movies through Question-Answering
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, June 2016.
Project page | arXiv preprint | Read the paper | Explore the data

Benchmark Data

The data is made available in simple JSON / text files for easy access in any environment. We provide Python scripts to help you get started by providing simple data loaders.

To obtain access to the stories, and evaluate new approaches on the test data, please register at our benchmark website.

Python data loader

import MovieQA
mqa = MovieQA.DataLoader()

Explore

Movies are indexed using their corresponding IMDb keys. For example
mqa.pprint_movie(mqa.movies_map['tt0133093'])

QAs are stored as a standard Python list
mqa.pprint_qa(mqa.qa_list[0])

Use

Get the list of movies in a particular split, use
movie_list = mqa.get_split_movies(split='train')

To get train or test splits of the QA along with a particular story, use
story, qa = mqa.get_story_qa_data('train', 'plot')

Supported splits are: train, val, test, full and story forms are: plot, subtitle, dvs, script

Video lists can be obtained per QA, or per movie using
vl_qa, _ = get_video_list('train', 'qa_clips') % per QA
vl_movie, _ = get_video_list('train', 'all_clips') % per movie

Build your own data/story loaders

We provide a simple interface to load all the data (QAs, movies) and stories through the code above. If you wish to modify something, you are welcome to use your own data loaders and access the raw data directly. The evaluation server submissions are simple text files (explained after login) and are independent of any data loaders.

qa.json

qid: A unique id for every question. Also indicates, train|val|test sets
imdb_key: The movie this question belongs to
question: The question string
answers: The five answer options
correct_index: Correct answer option (indexed by 0)
plot_alignment: split_plot file line numbers, to which this question corresponds
video_clips: Clips that are aligned with the question, to be used for answering

movies.json

imdb_key: A unique id for every movie. Corresponds to that used by IMDb
name: Movie title
year: Movie release year
genre: Movie genre classification
text: Text sources that are available for that movie

Data Releaselog

2017.01.14: Alignments between question and plot sentence, plot sentence and video clips
2016.11.08: Patch for 65 missing video clips
2016.09.10: Video meta-data released: Shot boundaries, frame-timestamp correspondence
2016.04.06: Removed missing video clips from qa.json
2016.03.30: v1.0 data release

Requirements

numpy
pysrt

youjiangxu / MovieQA_benchmark