YevhenKost / SemPrimsDetectionGA

Code to the Paper "Automatic Detection of Semantic Primitives Using Optimization Based on Genetic Algorithm"

Code to the Paper "Automatic Detection of Semantic Primitives Using Optimization Based on Genetic Algorithm"

Setup

1. Clone the repository:

git clone https://github.com/YevhenKost/SemPrimsDetectionGA.git

2. Install requirements

pip install -r requirements.txt

3. Fill the configs
   1. Pagerank model fitting parameters (conf/params_pagerank.json). The parameters description can be found via the following link: PageRank
   2. Word Vectorization paths and save names (conf/vectorization_configs.py). For each vectorizer provide required model paths on your local machine.

Usage

1. Prepare the dictionary in a following format and save to json file. Making a specific directory for the dictionary to store all the results is suggested. For example:

import json, os

# load dictionary
my_dict = {
    "cat": [
        {"definition": "a very cute animal"},
        {"definition": "makes muuuuuur"}
    ],
    "buy": [
        {"definition": "exchange something for a money"}
    ]
}

# save to the dir
SAVE_DIR = "cat_but_directory/"
os.makedirs(SAVE_DIR, exist_ok=True)
with open(os.path.join(SAVE_DIR, "dictionary.json"), "w") as f:
   json.dump(my_dict, f)

2. Convert dictionary to directed graph. It can be achieved via the command (paths are taken from the previous example):

python dict2graph.py --word_dictionary_path cat_but_directory/dictionary.json --stanza_dir LOADED_STANZA_MODELS/en --stanza_lang en --stop_words_lang english --save_dir cat_but_directory/ --drop_self_cycles true --lemm_always false

The arguments required:

• --word_dictionary_path: path to the dictionary saved in json format (see previous example)
• --stanza_dir: For lemmatization the stanza package is used. Can be "", than the stanza package will download everything based on the language, given in --stanza_lang. For model details, see Pipeline.
• --stanza_lang: Language of dictionary. List of available languages can be found here.
• --stop_words_lang: Stop words language to use. List of available languages can be found here.
• --save_dir: path to a directory, where the grapg dict files will be stored: word encoding dictionary and graph edges dictionary in json formats. Suggested approach is to use the same directory as for the graph.
• --drop_self_cycles: boolean, whether to delete the definitions, which have a word they suppose to define. For example, for the word "bark" the definition "to bark" will not be used during graph building.
• --lemm_always: boolean, whether to use lemmatization only if the word is not in dictionary vocabulary or always.
• --vocabulary_list_path (Optional): str, path to json with vocabulary to use for graph building. If None, the keys from file from word_dictionary_path will be used.
• --lemm_vocabulary (Optional): boolean, Ignored if vocabulary_list_path is empty. Whether to lemmatize words in vocabulary_list. The duplicates will be removed.

For more details:

python dict2graph.py -h

3. Run Generation of Permutation-based Semantic Primitives Sets:

python sp_generation.py --load_dir cat_but_directory/ --N 1000 --n_cores 12 --seed 2

Note, that it could take a while. For example, for a wordnet dictionary the generation of 1,000 SP lists took around a week with multiprocessing. The command execution will save in the --load_dir a generated lists in the following format and filename:

sp_sets_format = [
   [1,2,3], # sp set
   [10,2,5] # sp set
]

filename = f"candidates_{str(N)}_random{str(seed)}.json" # N and seed are taken from the arguments

The arguments required:

• --load_dir: path to directory, which contains graph.json file (generated on previous step). The generated SP lists will be saved here.
• --N: int, number of SP lists to generate (there is no gurantee that they will be all unqiue).
• --n_cores: int, how many cores to use during multiprocessing.
• --seed: int, fix random seed.

For more details:

python sp_generation.py -h

4. Fit PageRank model

python page_rank.py --load_dir cat_but_directory/ --fit_params_path conf/params_pagerank.json

The fitted model will be saved to --graph_path.

The arguments required:

• --load_dir: path to directory, which contains graph.json file (generated on the first step). In this directory the trained pagerank model will be saved.
• --fit_params_path: path to json file with pagerank parameters. See conf/params_pagerank.json

For more details:

python page_rank.py -h

5. Run algorithm

python run.py --load_dir cat_but_directory/ --sp_gen_lists_path cat_but_directory/candidates_1000_random2.json --n_threads 8 --val_prank_fill -1.0 --pop_size 100 --card_diff 50 --card_upper 2800 --save_dir GA_fit_model

Algorithm results will be saved to save_dir. See https://pymoo.org/interface/result.html. The decoded results will be stored in save_dir/sp_wordlists/.

The arguments required: * --load_dir: str, path to directory, which contains graph.json, encoding_dict.json and pagerank.pickle files (generated on previous steps). * --chp_path: str (optional), path to .npy checkpoint (if you want to continue training). After the model training this checkpoint will be saved in the save_dir. * --n_threads: int, number of cores to use for multiprocessing. * --sp_gen_lists_path: str, path to json file with stored generated SP lists (see step 3). * --val_prank_fill: negative float, value to use to return for mean pagerank objective function if the cycle is still detected in the graph. * --pop_size: int, population size (see [here](https://pymoo.org/algorithms/soo/ga.html#nb-ga)) * --card_diff: int, maximum possible cardinality deviation (constraint function: f(X) = (X - card_mean) ** 2 <= card_diff ** 2). * --card_mean: int, mean cardinality for the constraint (constraint function: f(X) = (X - card_mean) ** 2 <= card_diff). * --max_mutate: int, maximum number of elements to mutate per population. Default: 60. * --min_mutate: int, minimum number of elements to mutate per population. Default: 0. * --n_max_gen: int, maximum number of iterations to fit algorithm. Default: 30. * --save_dir: path, where training args, checkpoint and results will be stored.

For more details see:

python run.py -h

Testing

1. Prepare word lists
   Create a dir, where each word list should be in a text file with newline separated word

2. Fill up the preprocessing configs Before that fill up the conf/vectorization_configs.py and word_preprocessing_utils.py files. word_preprocessing_utils.py supports preprocessing for English, Spanish and Ukrainian at the moment, but it is possible to add new classes for other langs. In conf/vectorization_configs.py fill up the stemming/lemmatization fields with the suitable classes.

3. Vectorize target word lists

python vectorize_words.py --lists_dir wordlists/ --save_dir wordlists/embeddings/

The arguments required:

• --lists_dir: path to directory, which contains word lists (see Section 1).
• --save_dir: path, where the embeddings should be saved. Will generate a directory for each wordlist with the same name as file. In each dir the embeddings in .npy format will be saved.

4. Vectorize obtained word lists (see Section 5 of Usage)

python vectorize_words.py --lists_dir GA_fit_model/sp_wordlists --save_dir GA_fit_model/sp_embeddings/

5. Calculate and save metrics

python evaluate.py --pred_wordlist_embeddings_dir GA_fit_model/sp_embeddings --target_wordlist_dir wordlists/embeddings/ --save_dir GA_fit_model/ --metric cosine

The arguments required:

• --pred_wordlist_embeddings_dir: path to directory, where the embeddings for generated populations are stored (see previous step).
• --target_wordlist_dir: path to directory, where the embeddings for target word lists are stored (see step 3).
• --metric: metric to use. See https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html metric argument.
• --save_dir: path, where the metrics should be saved. The json file will be generated: metrics_metric.json, where metric is the specified one.