Repository for Skoltech Deep Learning course project "Investigation of topological losses for neural text generation" 2021
Daniil Cherniavskii, Saveliy Galochkin, Yehor Kravets, Lev Kosolapov, Ramilya Sharifullina, Dmitry Kryukov, Georgiy Kozhevniko
The code was test on Python 3.8.5; the requirements can be installed in a new enviroment with:
pip3 install -r requirements.txt
To finetune GPT-2 on ShortJokes dataset, as was done during the project, please refer to GPT-2 finetune notebook.
To examine the MST calculation functions we tried using, refer to mst.py.
Both default finetuned GPT-2 and the one trained with topological loss are localted here
Calculating original topological features is very time consuming, so we used an approximation to H0s, that is the sum of weights above threshold:
The final topological loss is defined as