Repository containing code and dataset of the paper "Do LLM Dream Of Ontologies?"
The repository is composed by 3 sub-directories:
Dataset
This directory contains the Gene and Uberon datasets. In particular:
- go_dataset.csv: it contains for each GO concept its ID, label, number of annotations in ontologies (not considered in this paper) and the number of web occurrences
- uberon_dataset.csv: it contains for each UBERON concept its ID, label, number of annotations in ontologies (not considered in this paper) and the number of web occurrences
- pythia12B_go_dataset_with_predictions_prompt2.csv: it contains the Pythia12B predictions (5th field) of the GO elements
- gpt-35-go_dataset_with_predictions.csv: it contains the GPT-3.5 predictions (5th field) of the GO elements
- gpt4-go_predictions.csv: it contains the GPT-4 predictions (5th field) of the GO elements
- sorted_by_occurrences_pythia12B_UBERON_dataset_prompt2.csv: it contains the Pythia12B predictions (5th field) of the UBERON elements
- sorted_by_occurrences_gpt35_uberon.csv: it contains the GPT-3.5 predictions (5th field) of the UBERON elements
- sorted_by_occurrences_gpt4_uberon.csv: it contains the GPT-4 predictions (5th field) of the UBERON elements
Perfromance_analysis
This directory contains the code to reproduce the results of the submitted paper. In particular:
- google_search_final.py: code to compute the number of web occurrences for a given <ID;label> string. Used to produce the datasets in the Dataset directory
- compute_performance.py: code to compute the correlation between performance and number of occurrences and to do the error Perfromance_analysis
- extract_k_most_predicted_labels.py: code to extract the k most predicted labels and to plot their distribution in buckets
- plot_dataset_statistics.py: code to plot the datasets statistics
- prompt_from_label_to_ID_GO_Pythia12B.py: Pythia prompt to GO
- prompt_from_label_to_ID_GO_GPT-35.py: GPT-3.5 prompt to GO
- prompt_from_label_to_ID_GO_GPT-4.py: GPT-4 prompt to GO
- prompt_from_label_to_ID_UBERON_Pythia12B.py: Pythia prompt to UBERON
- prompt_from_label_to_ID_UBERON_GPT-35.py: GPT-3.5 prompt to UBERON
- prompt_from_label_to_ID_UBERON_GPT-4.py: GPT-4 prompt to UBERON
Prompt invariance
This directory contains the code to reproduce the results of the Prompt Invariance proposed method. In particular:
- extracted_labels.csv: it contains random elements extracted from the GO
- prompt_invariance_experiments.py: code for prompting PI-1, PI-2 and PI-3
- compute_accuracy_per_bucket.py: code to plot AvPI and Accuracy per buckets
- evaluate_per_bucket.py: code to compute the prompt invariance
- 1_same_prompt_output.txt, 2_different_temperatures_output.txt, 3_different_languages_output.txt and SAMPLES_n_RANDOM_ELEMENTS_PER_BUCKETS.txt are output files