For our efficient metrics experiments we use the metrics
- BERTScore
- MoverScore
- XMoverScore
- SuperT
and replace the utilized BERT model with better performing and more efficient variants. The exact thoughts and approaches can be found in our report.
BERTScore, MoverScore, XMoverScore and SuperT can be found in their respective directories along with our received results. For KoBe and COMET it was not possible for us to get expressive results.
To experiment with BERTScore, go into its directory and open the BERTScore.ipynb
- To execute the file you need Google Drive and need to put the scores file into the specified directory from the variable
dpath
(/content/drive/MyDrive/data/MSE/scores_deen.csv
) - Upon executing the script will automatically iterate over the given variants and calculate their scores, time and memory usage and save a CSV file containing the results
To experiment with MoverScore, go into its directory and open the MoverScore_2_0.ipynb
- To execute the file you need Google Drive and need to put the scores file into the specified directory from the variable
dpath
(/content/drive/MyDrive/data/MSE/scores_deen.csv
) - Before executing, you need to specify the
model_name
so it can be used as an environment variable bymoverscore_v2
- Then the results of this variant will be saved as a CSV file
- when running the script multiple times with different
model_name
s, all resulting files can be merged with merge_scores, which will save one resulting CSV file comparing the variants
To experiment with XMoverScore and play around with different models, go into its directory and open XMoverScore/demo.py, where you find a modified script from the source folder.
- The attribute
variants
is a list with all transformers you want to test. If its a local model, you need to specify the source folder in the XMoverScore/scorer.py - script. If the transformer is onhuggingface.co
, an additional adjustement in XMoverScore/scorer.py could be necessary. - The results will be saved as a CSV file in the same folder.
- Plotting: To get the same plots from our report the XMoverScore/Datenaufbereitung.ipynb -notebook can be used. The values from the generated CSV-file will be used.
To experiment with SUPERT, open SUPERT.ipynb. There should be the code to execute all BERT variations with SUPERT
To get the other BERT variations running with the given SUPERT-Notebook:
- execute the cells, where the BERT-Models are downloaded
- Go to the cached download directory of the BERT models (on mac os:
~/.cache/torch/sentence_transformers/
) and open the folder of the downloaded models - Open
modules.json
and in line 6 rename....models.Transformer"
to....models.DistilBERT"
exemplary for DistilBERT (Write the correct model name depending of the BERT variation)
The correct naming of the model can be taken from the class names.