nianticlabs / rectified-features

[ECCV 2020] Single image depth prediction allows us to rectify planar surfaces in images and extract view-invariant local features for better feature matching

Single-Image Depth Prediction Makes Feature Matching Easier

Carl Toft, Daniyar Turmukhambetov, Torsten Sattler, Fredrik Kahl and Gabriel J. Brostow – ECCV 2020

Link to paper
Link to supplementary pdf

Code is coming soon...

Good local features improve the robustness of many 3D relocalization and multi-view reconstruction pipelines. The problem is that viewing angle and distance severely impact the recognizability of a local feature. Attempts to improve appearance invariance by choosing better local feature points or by leveraging outside information, have come with pre-requisites that made some of them impractical. In this paper, we propose a surprisingly effective enhancement to local feature extraction, which improves matching.

We show that CNN-based depths inferred from single RGB images are quite helpful, despite their flaws. They allow us to pre-warp images and rectify perspective distortions, to significantly enhance SIFT and BRISK features, enabling more good matches, even when cameras are looking at the same scene but in opposite directions.

Our pipeline finds planar patches according to estimated depth, and extracts features from rectified views of these patches. Non-rectified features are also extracted from regions that do not belong to planar patches.

💾 📸 Dataset

Dataset README

The "Strong Viewpoint Changes Dataset" is published as part of ECCV 2020 "Single-Image Depth Prediction Makes Feature Matching Easier" paper by Carl Toft, Daniyar Turmukhambetov, Torsten Sattler, Fredrik Kahl and Gabriel J. Brostow.

Please cite the paper if you are using this dataset.

The images, file pairs for evaluation and ground truth poses for the 8 scenes are available at:

https://storage.googleapis.com/niantic-lon-static/research/rectified-features/StrongViewpointChangesDataset/scene1.zip

https://storage.googleapis.com/niantic-lon-static/research/rectified-features/StrongViewpointChangesDataset/scene2.zip

https://storage.googleapis.com/niantic-lon-static/research/rectified-features/StrongViewpointChangesDataset/scene3.zip

https://storage.googleapis.com/niantic-lon-static/research/rectified-features/StrongViewpointChangesDataset/scene4.zip

https://storage.googleapis.com/niantic-lon-static/research/rectified-features/StrongViewpointChangesDataset/scene5.zip

https://storage.googleapis.com/niantic-lon-static/research/rectified-features/StrongViewpointChangesDataset/scene6.zip

https://storage.googleapis.com/niantic-lon-static/research/rectified-features/StrongViewpointChangesDataset/scene7.zip

https://storage.googleapis.com/niantic-lon-static/research/rectified-features/StrongViewpointChangesDataset/scene8.zip

The dataset is published with Attribution 4.0 International (CC BY 4.0) License, see: https://storage.googleapis.com/niantic-lon-static/research/rectified-features/StrongViewpointChangesDataset/LICENSE.txt

✏️ 📄 Citation

If you find our work useful or interesting, please consider citing our paper:

@inproceedings{toft-2020-rectified-features,
 title   = {Single-Image Depth Prediction Makes Feature Matching Easier},
 author  = {Carl Toft and
            Daniyar Turmukhambetov and
            Torsten Sattler and
            Fredrik Kahl and
            Gabriel J. Brostow
           },
 booktitle = {European Conference on Computer Vision ({ECCV})},
 year = {2020}
}

👩‍⚖️ License

Copyright © Niantic, Inc. 2020. Patent Pending. All rights reserved. Please see the license file for terms.