This repository contains dataset and DLD metrics analysis, and the research files of empirical study and investigation.
Our project is public at https://github.com/disensoftware/disentanglement-for-software
The state-of-art models and datasets are available as follow:
- BiLSTM: Author: Meheri et al., implemented by Liu et al.
- FF: Author: Kummerfeld et al., implemented by Kummerfeld et al.
- BERT: Author: Li et al., implemented by Liu et al.
- E2E: Author: Liu et al., implemented by Liu et al.
- PtrNet: Author: Yu et al., implemented by Yu et al.
Our dataset is constructed with two different formats: original and json. Various data formats are intended for different model inputs.
Moreover, we introduce the addresses of traditional symbolic datasets for referencing and analyzing.
The source code of traditional metric is available in utils.py, including NMI, ARI and Shen-F.
The F1 and DLD require the reformation of predicted data. Since the preprocessing data of intermediate steps are private for all works, we only provide the 1st version of reformat code reformat_e2d_source_data.py for reference without bug fixing.
If you need the complete data of annotation and preprocessing, please contact us via (OUR EMAIL ADDRESS)
We provide the experiment data of each models and metrics in experiment fold.
- Data of models comparison experiment: model_comparison.
- Data of metrics comparison experiment: metric_comparison.
The performance of DLD is shown as follows:
| Analysis | RMSE | MAE | PEA | IST | PST | ANOVA |
|---|---|---|---|---|---|---|
| NMI | 0.38 | 0.34 | 0.08 | e-55 | e-46 | e-55 |
| ARI | 0.37 | 0.32 | 0.02 | e-19 | e-17 | e-19 |
| Shen-F | 0.41 | 0.36 | 0.17 | e-69 | e-59 | e-69 |
| F1 | 0.19 | 0.14 | 0.85 | e-4 | e-14 | e-4 |
| DLD | 0.08 | 0.07 | 0.92 | 0.51 | 0.31 | 0.51 |
All diagrams of article are available in diagrams directory.
- Performances: The original and retraining model performances.
- Metrics comparison: The prove of DLD superiority.
- Bad Cases: The card sorted results of incorrect disentanglement.