This is the repository for the paper "One Step Further: Evaluating Interpreters Using Metamorphic Testing" submitted to ISSTA 2022. The systematical evaluation and metamorphic testing for interpretation methods are implemented here.
All original images used in our experiment are from the COCO2017 dataset. The official provides a large number of data annotations and API interfaces to process images.
Folder "backdoor" shows the strategy of training a backdoor model by updating the existing model incrementally. The training process is realized in backdoor_retrain.py.
Folder "interpretation_methods" shows seven methods for metamorphic testing. Interpretation methods except LIME are realized by using TorchRay. We realize LIME by its own individual implementation. Each code is used to evaluate the interpretation results of original images and metamorphic images.
The folder "metamorphic_technologies" shows three technologies to generate metamorphic images. To generate diverse and natural-looking sets of images as inputs of metamorphic testing, we use MetaOD, a streamlined workflow that performs object extraction, object refinement/selection, and object insertion to synthesize metamorphic images in an efficient and adaptive manner. Code preproess_val.py can realize the main process of inserting an object, whose object extraction module uses YOLACT to realize.
We use deepfill v2 to perform image inpainting. First, we need to train an image inpainting model with the COCO2017 dataset based on the deepfill v2 algorithm. And then, we input mask and mask image into the trained image inpainting model to generate the result of image inpainting, which is natural-looking.
"Models" published in releases contains pre-trained normal model, backdoor model, and models used in inserting the object, deleting the object (image inpainting).
The experimental results are massive, and we first put some results images of our paper on the folder "results". More results can be found in Google Drive.