Performance calculation tool for Hateful Memes Challenge
This simple project uses simple functions from pretty errors, click, pandas, sklearn. Therefore, one can go to these link and install as instructions. This works with Python 3.7
calc_test.py calculates AUC ROC and Accuracy scores. One can run python calc_test.py --help for instruction or can read the source code. It's very simple.
calc_test.py takes test_seen.jsonl (Phase 1) or test_unseen.jsonl (Phase 2) and result.csv. Importantly, test_seen.jsonl, test_unseen.jsonl must have labels.
result.csv must have to three columns:
- Meme identification number,
id - Probability that the meme is hateful,
proba(must be a float) - Binary label that the meme is hateful (
1) or non-hateful (0),label(must be an int)
combine.py is meant to combine all train.jsonl, dev_seen.jsonl, dev_unseen.jsonl, test_seen.jsonl, test_unseen.jsonl into data_test.jsonl. Importantly, test_seen.jsonl, test_unseen.jsonl must have labels. By combining so, data_test.jsonl contains all metadata of all memes in the dataset.