Multimodal Semantic Mismatch Detection in Social Media Posts
Kehan Wang · Seth Z. Zhao · David Chan · Avideh Zakhor · John Canny
IEEE 24th International Workshop on Multimedia Signal Processing (MMSP) 2022
Download Dataset
We release two dataset, Twitter-1M and Twitter-60k-with-audio, with precomputed features from S3D on video and DeBERTa-v3 on text. Twitter-60k-with-audio also contains DeBERTa-v3 features on the transcript of tweet audios, transcribed using Wav2vec 2.0.
- Twitter-1M contains all features in a zip, and their corresponding tweet-ids. Tweet ids and features are in the same order. Features can be loaded through
np.load. - Twitter-60k-with-audio contains all features and ids in a
dictsaved in.npyfiles. To load the dict, use
import numpy as np
dictionary = np.load(f'60k_{mode}_features.npy', allow_pickle=True).flatten()[0]
dictionary.keys() # dict_keys(['text_feats', 'caption_feats', 'video_feats', 'ids'])Acknowledgement
We would like to graciously acknowledge Google for partially providing cloud computing resouces for this project, and Twitter for Academic Research API.
Citation
Please cite Semantic Mismatch Detector in your publications if it helps your research:
@inproceedings{SMD,
title = {Multimodal Semantic Mismatch Detection in Social Media Posts},
author = {Kehan Wang and Seth Z. Zhao and David Chan and Avideh Zakhor and John Canny},
booktitle = {Proceedings of IEEE 24th International Workshop on Multimedia Signal Processing (MMSP)},
year = {2022}
}