vignywang / Awesome-Local-Feature-Matching

Awesome-Local-Feature-Matching

This is the official repository for the Information Fusion 2024 paper "Local feature matching using deep learning: A survey".

Abstract

Local feature matching enjoys wide-ranging applications in the realm of computer vision, encompassing domains such as image retrieval, 3D reconstruction, and object recognition. However, challenges persist in improving the accuracy and robustness of matching due to factors like viewpoint and lighting variations. In recent years, the introduction of deep learning models has sparked widespread exploration into local feature matching techniques. The objective of this endeavor is to furnish a comprehensive overview of local feature matching methods. These methods are categorized into two key segments based on the presence of detectors. The Detector-based category encompasses models inclusive of Detect-then-Describe, Joint Detection and Description, Describe-then-Detect, as well as Graph Based techniques. In contrast, the Detector-free category comprises CNN Based, Transformer Based, and Patch Based methods. Our study extends beyond methodological analysis, incorporating evaluations of prevalent datasets and metrics to facilitate a quantitative comparison of state-of-the-art techniques. The paper also explores the practical application of local feature matching in diverse domains such as Structure from Motion, Remote Sensing Image Registration, and Medical Image Registration, underscoring its versatility and significance across various fields. Ultimately, we endeavor to outline the current challenges faced in this domain and furnish future research directions, thereby serving as a reference for researchers involved in local feature matching and its interconnected domains. A comprehensive list of studies in this survey is available at https://github.com/vignywang/Awesome-Local-Feature-Matching.

News

😊 This project is under development. You can hit the STAR and WATCH to follow the updates.

• 2024/03/03: Our Awesome-Local-Feature-Matching reasoning paper: Local feature matching using deep learning: A survey has been accepted by Information Fusion 2024.

Table of Contents

• Awesome-Local-Feature-Matching
  • Abstract
  • News
  • Table of Contents
  • Related Surveys
  • Detector-based Models
    • Detect-then-Describe
    • Joint Detection and Description
    • Describe-then-Detect
    • Graph Based
  • Detector-free Models
    • CNN Based
    • Transformer Based
    • Patch Based
  • Applications
    • Structure from Motion
    • Remote Sensing Image registration
    • Medical Image Registration
  • Acknowledgement

Related Surveys

• Advances and Challenges in Multimodal Remote Sensing Image Registration (IEEE, 2023) [paper]
• A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond (Arxiv, 2023) [paper]
• Image Feature Information Extraction for Interest Point Detection: A Comprehensive Review (IEEE, 2022) [paper]
• Challenges in Image Matching for Cultural Heritage: An Overview and Perspective (ICIAP, 2022) [paper]
• A review of multimodal image matching: Methods and applications (Information Fusion, 2021) [paper]
• Image Matching from Handcrafted to Deep Features: A Survey (IJCV, 2021) [paper]
• Recent advances in local feature detector and descriptor: a literature survey (IJMIR, 2020) [paper]
• Local feature descriptor for image matching: A survey (IEEE, 2018) [paper]

Detector-based Models

Fig.1: The comparison of various prominent Detector-based pipelines for trainable local feature matching is presented. Here, the categorization is based on the relationship between the detection and description steps: (a) Detect-then-Describe framework, (b) Joint Detection and Description framework, and (c) Describe-then-Detect framework.

Detect-then-Describe

• Zippypoint: Fast interest point detection, description, and matching through mixed precision discretization (CVPR, 2023) [paper]
• Attention weighted local descriptors (IEEE, 2023) [paper]
• Mtldesc: Looking wider to describe better (AAAI, 2022) [paper]
• Cndesc: Cross normalization for local descriptors learning (IEEE, 2022) [paper]
• Alike: Accurate and lightweight keypoint detection and descriptor extraction (IEEE, 2022) [paper]
• Neural outlier rejection for self-supervised keypoint learning (ICLR, 2020) [paper]
• Hynet: Learning local descriptor with hybrid similarity measure and triplet loss (NIPS, 2020) [paper]
• Key. net: Keypoint detection by handcrafted and learned cnn filters (ICCV, 2019) [paper]
• Beyond cartesian representations for local descriptors (ICCV, 2019) [paper]
• Sosnet: Second order similarity regularization for local descriptor learning (CVPR, 2019) [paper]
• Geodesc: Learning local descriptors by integrating geometry constraints (ECCV, 2018) [paper]
• Working hard to know your neighbor’s margins: Local descriptor learning loss (NIPS, 2017) [paper]
• L2-net: Deep learning of discriminative patch descriptor in euclidean space (CVPR, 2017) [paper]
• Learning to assign orientations to feature points (CVPR, 2016) [paper]

Joint Detection and Description

• Sfd2: Semantic-guided feature detection and description (CVPR, 2023) [paper]
• Learning semantic-aware local features for long term visual localization (IEEE, 2022) [paper]
• Learning of low-level feature keypoints for accurate and robust detection (WACV, 2021) [paper]
• Aslfeat: Learning local features of accurate shape and localization (CVPR, 2020) [paper]
• Mlifeat: Multi-level information fusion based deep local features (ACCV, 2020) [paper]
• Hdd-net: Hybrid detector descriptor with mutual interactive learning (ACCV, 2020) [paper]
• Reinforced feature points: Optimizing feature detection and description for a high-level task (CVPR, 2020) [paper]
• Rfnet: An end-to-end image matching network based on receptive field (CVPR, 2019) [paper]
• Lf-net: Learning local features from images (NIPS, 2018) [paper]

Describe-then-Detect

• Shared coupling-bridge for weakly supervised local feature learning (Arxiv, 2022) [paper]
• Redfeat: Recoupling detection and description for multimodal feature learning (IEEE, 2022) [paper]
• Decoupling makes weakly supervised local feature better (CVPR, 2022) [paper]
• D2d: Keypoint extraction with describe to detect approach (ACCV, 2020) [paper]

Graph Based

Fig.2: General GNN Matching Model Architecture. Firstly, keypoint positions p_i along with their visual descriptors d_i are mapped into individual vectors. Subsequently, self-attention layers and cross-attention layers are thereafter applied alternately, L times, within a graph neural network to create enhanced matching descriptors. Finally, the Sinkhorn Algorithm is utilized to determine the optimal partial assignment.

• Gluestick: Robust image matching by sticking points and lines together (ICCV, 2023) [paper]
• Learning feature matching via matchable keypoint-assisted graph neural network (Arxiv, 2023) [paper]
• Lightglue: Local feature matching at light speed (ICCV, 2023) [paper]
• Htmatch: An efficient hybrid transformer based graph neural network for local feature matching (Signal Processing, 2023) [paper]
• Paraformer: Parallel attention transformer for efficient feature matching (Arxiv, 2023) [paper]
• Resmatch: Residual attention learning for local feature matching (Arxiv, 2023) [paper]
• Densegap: graph-structured dense correspondence learning with anchor points (ICPR, 2022) [paper]

Detector-free Models

Acknowledgement

This work is supported by Beijing Natural Science Foundation No. JQ23014, in part by the National Natural Science Foundation of China (No. 62271074).