IGNet

Learning a Graph Neural Network with Cross Modality Interaction for Image Fusion

in Proceedings of the 31st ACM International Conference on Multimedia (ACM MM 2023)
by Jiawei Li, Jiansheng Chen, JinyuanLiu and Huimin Ma

Overall performance comparison:

Framework of our proposed IGNet:

GIF Demo:

Requirements

python 3.7

torch 1.7.0

torchvision 0.8.0

opencv 4.5

numpy 1.21.6

pillow 9.4.0

Dataset setting

We give 5 test image pairs as examples in [TNO], [MFNet] and [M3FD] datasets, respectively.

Moreover, you can set your own test datasets of different modalities under ./test_images/..., like:
test_images
├── ir
|   ├── 1.png
|   ├── 2.png
|   └── ...
├── vis
|   ├── 1.png
|   ├── 2.png
|   └── ...
Note that if ./test_images/vis/xxx.png is in single-channel L format, you should use LtoRGB.py to convert it to three-channel RGB format.

Test

The pre-trained model has given in ./model/IGNet.pth. Please run test.py to get fused results, and you can check them in:
results
├── 1.png
├── 2.png
└── ...

Experimental results

We compared our proposed IGNet with [DIDFuse], [U2Fusion], [SDNet], [TarDAL], [UMFusion], [DeFusion] and [ReCoNet].

Fusion results:

After retaining the fusion results of all methods on [YOLOv5] and [DeepLabV3+], we compare the corresponding detection and segmentation results with IGNet.

Detection & Segmentation results:

Please refer to the paper for more experimental results and details.

Citation

@inproceedings{li2023learning,
   title = {Learning a Graph Neural Network with Cross Modality Interaction for Image Fusion},
   author = {Li, Jiawei and Chen, Jiansheng and Liu, Jinyuan and Ma, Huimin},
   booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
   pages = {4471–4479},
   year = {2023},
}

Realted works

Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang and Nikola K. Kasabov. GeSeNet: A General Semantic-guided Network with Couple Mask Ensemble for Medical Image Fusion. IEEE Transactions on Neural Networks and Learning Systems (IEEE TNNLS), 2023. [Paper] [Code]

Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang and Nikola K. Kasabov. Learning a Coordinated Network for Detail-refinement Multi-exposure Image Fusion. IEEE Transactions on Circuits and Systems for Video Technology (IEEE TCSVT), 2022, 33(2): 713-727. [Paper]

Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang and Nikola K. Kasabov. Infrared and visible image fusion based on residual dense network and gradient loss. Infrared Physics & Technology, 2023, 128: 104486. [Paper]

Jia Lei, Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang and Nikola K. Kasabov. GALFusion: Multi-exposure Image Fusion via a Global-local Aggregation Learning Network. IEEE Transactions on Instrumentation and Measurement (IEEE TIM), 2023, 72: 1-15. [Paper] [Code]

Acknowledgement

Our Code is partially adapted from [Cas-Gnn]. Please refer to their excellent work for more details.

Contact

If you have any questions, please create an issue or email to me (Jiawei Li).

lok-18 / IGNet