Introduction
Trex is a tool to match semantically similar functions based on transfer learning. It extends Roberta encoder with masked language modeling objective [1] by supporting multi-field inputs and outputs.
Installation
We recommend conda to setup the environment and install the required packages.
First, create the conda environment,
conda create -n trex python=3.7 numpy scipy scikit-learn
and activate the conda environment:
conda activate trex
Then, install the latest PyTorch (assume you have GPU):
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
Finally, enter the trex root directory: e.g., path/to/trex, and install trex:
pip install --editable .
Preparation
Pretrained models:
Create the checkpoints and checkpoints/pretrain subdirectory in path/to/trex
mkdir checkpoints, mkdir checkpoints/pretrain
Download our pretrained weight parameters and put in checkpoints/pretrain
Sample data for finetuning similarity
We provide the sample training/testing files of finetuning in data-src/clr_multifield_any
If you want to prepare the finetuning data yourself, make sure you follow the format shown in data-src/clr_multifield_any (coming soon: tokenization script).
We have to binarize the data to make it ready to be trained. To binarize the training data for finetuning, run:
python command/clr_multifield/preprocess_any.py
The binarized training data ready for finetuning (for detecting similarity) will be stored at data-bin/clr_multifield_any
Training
To finetune the model, run:
./command/clr_multifield/finetune_any.sh
The scripts loads the pretrained weight parameters from checkpoints/pretrain/ and finetunes the model.
Sample data for pretraining on micro-traces
We also provide (10K) samples and scripts to demonstrate how to pretrain the model. To binarize the training data for pretraining, run:
python command/roberta_multifield/preprocess_pretrain_10k.py
The binarized training data ready for pretraining will be stored at data-bin/pretrain_10k
To pretrain the model, run:
./command/roberta_multifield/pretrain_10k.sh
The pretrained model will be checkpointed at checkpoints/pretrain_10k
Dataset
We put our dataset here.
References
[1] Liu, Yinhan, et al. "Roberta: A robustly optimized bert pretraining approach." arXiv preprint arXiv:1907.11692 (2019).