hanna-xu / MURF

MURF

Code for "MURF: Mutually Reinforcing Multi-modal Image Registration and Fusion" (IEEE TPAMI 2023).

Recommended Environment:

python=3.6
tensorflow-gpu=1.14.0
numpy=1.19
scikit-image=0.17.2
pillow=8.2

Task #1: Shared information extraction

To train:

• Download the training data:
  • RGB-IR (created with RoadScene dataset)
  • RGB-NIR (created with VIS-NIR Scene dataset)
  • PET-MRI (created with Harvard dataset)
  • CT-MRI (created with Harvard dataset)
  • or create your training dataset according to it
    
• Run CUDA_VISIBLE_DEVICES=0 python main.py

To test:

• Put the test data in ./test_imgs/
  
• Run CUDA_VISIBLE_DEVICES=0 python test.py
  

Task #2: Multi-scale coarse registration

• This task is based on Task #1, so the code and models in task #1 should be downloaded and prepared in advance.

To train:

• Download the training data: RGB-IR, RGB-NIR, PET-MRI, CT-MRI or create your training dataset.
• Adjust task1_model_path in main.py to the path where you store the model in task #1.
• Run CUDA_VISIBLE_DEVICES=0,1 python main.py
  

In some tasks:

• Put more large-resolution training images in ./large_images_for_training/
• Finetune the trained model with large-resolution images by running CUDA_VISIBLE_DEVICES=0,1 python finetuning.py

To test:

• Prepare test data (one of the two ways):
  • Put the test images in ./test_data/images/ or
  • Put the test data (including images and landmark) in ./test_data/LM/ in .mat format
    
• Run test code:
  • CUDA_VISIBLE_DEVICES=0 python test.py or
  • CUDA_VISIBLE_DEVICES=0,1 python test.py or
  • CUDA_VISIBLE_DEVICES=0,1 python test_w_finetuning.py

Task #3: Fine registration and fusion

To train:

• Download the training data (same as that in Task #1 and the non-rigid deformation is applied subsequently)
• Run CUDA_VISIBLE_DEVICES=0 python main.py

In some tasks:

• Put more large-resolution training images in ./large_images_for_training/
• Finetune the trained model with large-resolution images by running CUDA_VISIBLE_DEVICES=0 python finetuning.py

To test:

• Put the test data in ./test_imgs/
  
• Run CUDA_VISIBLE_DEVICES=0 python test.py
  

The previous version of this work:

@inproceedings{xu2022rfnet,
  title={Rfnet: Unsupervised network for mutually reinforcing multi-modal image registration and fusion},
  author={Xu, Han and Ma, Jiayi and Yuan, Jiteng and Le, Zhuliang and Liu, Wei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={19679--19688},
  year={2022}
}