KangLiao929 / Awesome-Deep-Camera-Calibration

Deep Learning for Camera Calibration and Beyond: A Survey

Awesome-Deep-Camera-Calibration

Popular calibration objectives, models, and extended applications in camera calibration

Our repository provides:

1) a time line of the development in the related research community 2) a taxonomy of the survey 3) a public and holistic benchmark 4) collects deep learning-based camera calibration Publication, Objective, Network 5) dataset summary

More content and details can be found in our Survey Paper: Deep Learning for Camera Calibration and Beyond: A Survey.

📝 Changelog/News

• 2023.03.19: The survey of the arXiv version is online.
• 2024.01.05: The benchmark is released. Please refer to the dataset link and more details in Benchmark.
• 2024.06.04: More details about the calibrated camera parameters in our Benchmark are updated.
• 2024.06.05: Update the survey paper (Section 3.4) with more technical discussions with NeRF, especially in the camera parameter initialization.
• 2024.06.05: Update the survey paper (supplementary material) with the evaluation on the constructed benchmark.
• Update the literature reviews for 2023 and 2024.

🚩Contents

1. Basics
2. Taxonomy
3. Benchmark
4. Methods
5. Dataset
6. Citation
7. 知乎导读

🚩Basics

• Multiple view geometry in computer vision - Hartley, R., & Zisserman, A. (2004)
• A Flexible New Technique for Camera Calibration - Zhengyou Zhang. (2000)

🚩Taxonomy

The structural and hierarchical taxonomy of camera calibration with deep learning. Some classical methods are listed under each category.

A concise milestone of deep learning-based camera calibration methods.

We classify all methods based on the uncalibrated camera model and its extended applications: standard model, distortion model, cross-view model, and cross-sensor model.

🚩Benchmark

Overview of our collected benchmark, which covers all models reviewed in this survey.

In this dataset, the image and video were derived from diverse cameras under different environments. The accurate ground truth and label are provided for each data. Please refer to the dataset link and more details in Benchmark.

🚩Methods

Year	Publication	Title	Abbreviation	Objective	Platform	Network
2015	ICIP	Deepfocal: A method for direct focal length estimation	DeepFocal	Intrinsics	Caffe	AlexNet
2015	ICCV	Posenet: A convolutional network for real-time 6-dof camera relocalization	PoseNet	Extrinsics	Caffe	GoogLeNet
2016	BMVC	Horizon lines in the wild	DeepHorizon	Extrinsics	Caffe	GoogLeNet
2016	CVPR	Detecting vanishing points using global image context in a non-manhattan world	DeepVP	Extrinsics	Caffe	AlexNet
2016	ACCV	Radial lens distortion correction using convolutional neural networks trained with synthesized images	Rong et al.	Distortion coefficients	Caffe	AlexNet
2016	RSSW	Deep image homography estimation	DHN	Projection matrixs	Caffe	VGG
2017	CVPR	Clkn: Cascaded lucas-kanade networks for image alignment	CLKN	Projection matrixs	Torch	CNN + Lucas-Kanade layer
2017	ICCVW	Homography estimation from image pairs with hierarchical convolutional networks	HierarchicalNet	Projection matrixs	TensorFlow	VGG
2017	CVPR	Unrolling the Shutter: CNN to Correct Motion Distortions	URS-CNN	Undistortion	Torch	CNNs
2017	IV	RegNet: Multimodal sensor registration using deep neural networks	RegNet	Camera + LiDAR	Caffe	CNNs
2018	CVPR	A perceptual measure for deep single image camera calibration	Hold-Geoffroy et al.	Intrinsics + Extrinsics		DenseNet
2018	CVMP	DeepCalib: a deep learning approach for automatic intrinsic calibration of wide field-of-view cameras	DeepCalib	Intrinsics + Distortion coefficients	TensorFlow	Inception-V3
2018	ECCV	Fisheyerecnet: A multi-context collaborative deep network for fisheye image rectification	FishEyeRecNet	Distortion coefficients	Caffe	VGG
2018	ICPR	Radial lens distortion correction by adding a weight layer with inverted foveal models to convolutional neural networks	Shi et al.	Distortion coefficients	PyTorch	ResNet
2018	ECCV	Deep fundamental matrix estimation	DeepFM	Projection matrixs	PyTorch	ResNet
2018	ECCVW	Deep fundamental matrix estimation without correspondences	Poursaeed et al.	Projection matrixs		CNNs
2018	RAL	Unsupervised deep homography: A fast and robust homography estimation model	UDHN	Projection matrixs	TensorFlow	VGG
2018	ACCV	Rethinking planar homography estimation using perspective fields	PFNet	Projection matrixs	TensorFlow	FCN
2018	IROS	CalibNet: Geometrically Supervised Extrinsic Calibration using 3D Spatial Transformer Networks	CalibNet	Camera + LiDAR	TensorFlow	ResNet
2018	ICRA	DeepVP: Deep Learning for Vanishing Point Detection on 1 Million Street View Images	Chang et al.	Standard	Matconvnet	AlexNet
2019	CVPR	Deep single image camera calibration with radial distortion	Lopez et al.	Intrinsics + Extrinsics + Distortion coefficients	PyTorch	DenseNet
2019	ICCV	UprightNet: geometry-aware camera orientation estimation from single images	UprightNet	Extrinsics	PyTorch	U-Net
2019	IROS	Degeneracy in self-calibration revisited and a deep learning solution for uncalibrated slam	Zhuang et al.	Intrinsics + Distortion coefficients	PyTorch	ResNet
2019	PRL	Self-Supervised deep homography estimation with invertibility constraints	SSR-Net	Projection matrixs	PyTorch	ResNet
2019	ICCVW	A geometric approach to obtain a bird's eye view from an image	Abbas et al.	Projection matrixs	TensorFlow	CNNs
2019	TCSVT	DR-GAN: Automatic radial distortion rectification using conditional GAN in real-time	DR-GAN	Undistortion	TensorFlow	GANs
2019	TCSVT	Distortion rectification from static to dynamic: A distortion sequence construction perspective	STD	Undistortion	TensorFlow	GANs
2019	VR	Deep360Up: A deep learning-based approach for automatic VR image upright adjustment	Deep360Up	Extrinsics		DenseNet
2019	JVCIR	Unsupervised fisheye image correction through bidirectional loss with geometric prior	UnFishCor	Distortion coefficients	TensorFlow	VGG
2019	CVPR	Blind geometric distortion correction on images through deep learning	BlindCor	Undistortion	PyTorch	U-Net
2019	CVPR	Learning structure-and-motion-aware rolling shutter correction	RSC-Net	Undistortion	PyTorch	ResNet
2019	CVPR	Learning to calibrate straight lines for fisheye image rectification	Xue et al.	Distortion coefficients	PyTorch	ResNet
2019	ICCV	Learning perspective undistortion of portraits	Zhao et al.	Intrinsics + Undistortion		VGG + U-Net
2019	NeurIPS	NeurVPS: Neural Vanishing Point Scanning via Conic Convolution	NeurVPS	Standard	PyTorch	CNNs
2020	CVPR	End-to-end camera calibration for broadcast videos	Sha et al.	Projection matrixs	TensorFlow	Siamese-Net + U-Net
2020	ECCV	Neural geometric parser for single image camera calibration	Lee et al.	Intrinsics + Extrinsics		PointNet + CNNs
2020	ICRA	Learning camera miscalibration detection	MisCaliDet	Average pixel position difference	TensorFlow	CNNs
2020	WACV	DeepPTZ: deep self-calibration for PTZ cameras	DeepPTZ	Intrinsics + Extrinsics + Distortion coefficients	PyTorch	Inception-V3
2020	CVPR	Deep homography estimation for dynamic scenes	MHN	Projection matrixs	TensorFlow	VGG
2020	ACMMM	SRHEN: Stepwise-Refining Homography Estimation Network via Parsing Geometric Correspondences in Deep Latent Space	SRHEN	Projection matrixs		CNNs
2020	ECCV	360∘ camera alignment via segmentation	Davidson et al.	Extrinsics		FCN
2020	ECCV	Content-aware unsupervised deep homography estimation	CA-UDHN	Projection matrixs	PyTorch	FCN + ResNet
2020	IROS	Deep keypoint-based camera pose estimation with geometric constraints	DeepFEPE	Extrinsics	PyTorch	VGG + PointNet
2020	TIP	Model-free distortion rectification framework bridged by distortion distribution map	DDM	Undistortion	Tensorflow	GANs
2020	TIP	Deep face rectification for 360° dual-fisheye cameras	Li et al.	Undistortion		CNNs
2020	ICPR	Position-aware and symmetry enhanced GAN for radial distortion correction	PSE-GAN	Undistortion		GANs
2020	ICIP	A simple yet effective pipeline for radial distortion correction	RDC-Net	Undistortion	PyTorch	ResNet
2020	ICASSP	Self-supervised deep learning for fisheye image rectification	FE-GAN	Undistortion	PyTorch	GANs
2020	CVPR	RDCFace: radial distortion correction for face recognition	RDCFace	Undistortion		ResNet
2020	arXiv	Fisheye distortion rectification from deep straight lines	LaRecNet	Distortion coefficients	PyTorch	ResNet
2020	CVPR	Height and uprightness invariance for 3d prediction from a single view	Baradad et al.	Intrinsics + Extrinsics	PyTorch	CNNs
2020	CVPR	What does plate glass reveal about camera calibration?	Zheng et al.	Intrinsics + Extrinsics		CNNs
2020	ECCV	Single view metrology in the wild	Zhu et al.	Intrinsics + Extrinsics	PyTorch	CNNs + PointNet
2020	CVPR	Deep Shutter Unrolling Network	DeepUnrollNet	Undistortion	PyTorch	FCN
2020	RAL	RGGNet: Tolerance Aware LiDAR-Camera Online Calibration With Geometric Deep Learning and Generative Model	RGGNet	Camera + LiDAR	Tensorflow	ResNet
2020	IROS	CalibRCNN: Calibrating Camera and LiDAR by Recurrent Convolutional Neural Network and Geometric Constraints	CalibRCNN	Camera + LiDAR	Tensorflow	RNN
2020	ICRA	Online Camera-LiDAR Calibration with Sensor Semantic Information	SSI-Calib	Camera-LiDAR	Tensorflow	CNNs
2020	arXiv	SOIC: Semantic Online Initialization and Calibration for LiDAR and Camera	SOIC	Camera-LiDAR	-	ResNet+PointRCNN
2020	ICPR	NetCalib: A Novel Approach for LiDAR-Camera Auto-calibration Based on Deep Learning	NetCalib	Camera-LiDAR	PyTorch	CNNs
2021	TCI	Online training of stereo self-calibration using monocular depth estimation	StereoCaliNet	Extrinsics	PyTorch	U-Net
2021	ICCV	CTRL-C: Camera calibration TRansformer with Line-Classification	CTRL-C	Intrinsics + Extrinsics	PyTorch	Transformer
2021	ICCVW	Deep single fisheye image camera calibration for over 180-degree projection of field of view	Wakai et al.	Intrinsics + Extrinsics		DenseNet
2021	TIP	A deep ordinal distortion estimation approach for distortion rectification	OrdianlDistortion	Distortion coefficients	TensorFlow	CNNs
2021	TCSVT	Revisiting radial distortion rectification in polar-coordinates: A new and efficient learning perspective	PolarRecNet	Undistortion	PyTorch	VGG + U-Net
2021	PRL	DQN-based gradual fisheye image rectification	DQN-RecNet	Undistortion	PyTorch	VGG
2021	CVPR	Practical wide-angle portraits correction with deep structured models	Tan et al.	Undistortion	PyTorch	U-Net
2021	CVPR	Progressively complementary network for fisheye image rectification using appearance flow	PCN	Undistortion	PyTorch	U-Net
2021	ICCV	Multi-level curriculum for training a distortion-aware barrel distortion rectification model	DaRecNet	Undistortion	TensorFlow	U-Net
2021	CVPR	Deep Lucas-Kanade homography for multimodal image alignment	DLKFM	Projection matrixs	TensorFlow	Siamese-Net
2021	ICCV	LocalTrans: A multiscale local transformer network for cross-resolution homography estimation	LocalTrans	Projection matrixs	PyTorch	Transformer
2021	ICCV	Motion basis learning for unsupervised deep homography estimation with subspace projection	BasesHomo	Projection matrixs	PyTorch	ResNet
2021	ICIP	Fast and accurate homography estimation using extendable compression network	ShuffleHomoNet	Projection matrixs	TensorFlow	ShuffleNet
2021	TCSVT	Depth-aware multi-grid deep homography estimation with contextual correlation	DAMG-Homo	Projection matrixs	TensorFlow	CNNs
2021	BMVC	A simple approach to image tilt correction with self-attention MobileNet for smartphones	SA-MobileNet	Extrinsics	TensorFlow	MobileNet
2021	ICCV	SPEC: Seeing people in the wild with an estimated camera	SPEC	Intrinsics + Extrinsics	PyTorch	ResNet
2021	CVPR	Wide-Baseline Relative Camera Pose Estimation with Directional Learning	DirectionNet	Extrinsics	TensorFlow	U-Net
2021	CVPR	Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes	JCD	Undistortion	PyTorch	FCN
2021	CVPRW	LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network	LCCNet	Camera + LiDAR	PyTorch	CNNs
2021	Sensors	CFNet: LiDAR-Camera Registration Using Calibration Flow Network	CFNet	Camera + LiDAR	PyTorch	FCN
2021	ICCV	Inverting a Rolling Shutter Camera: Bring Rolling Shutter Images to High Framerate Global Shutter Video	Fan et al.	Distortion	PyTorch	U-Net
2021	ICCV	SUNet: Symmetric Undistortion Network for Rolling Shutter Correction	SUNet	Distortion	PyTorch	DenseNet+ResNet
2021	IROS	SemAlign: Annotation-Free Camera-LiDAR Calibration with Semantic Alignment Loss	SemAlign	Camera-LiDAR	PyTorch	CNNs
2022	CVPR	Deep vanishing point detection: Geometric priors make dataset variations vanish	DVPD	Extrinsics	PyTorch	CNNs
2022	ICRA	Self-supervised camera self-calibration from video	Fang et al.	Intrinsics + Extrinsics	PyTorch	CNNs
2022	ICASSP	Camera calibration through camera projection loss	CPL	Intrinsics + Extrinsics	TensorFlow	Inception-V3
2022	CVPR	Iterative Deep Homography Estimation	IHN	Projection matrixs	PyTorch	Siamese-Net
2022	CVPR	Unsupervised Homography Estimation with Coplanarity-Aware GAN	HomoGAN	Projection matrixs	PyTorch	GANs
2022	CVPR	Semi-Supervised Wide-Angle Portraits Correction by Multi-Scale Transformer	SS-WPC	Undistortion	PyTorch	Transformer
2022	CVPR	Learning Adaptive Warping for Real-World Rolling Shutter Correction	AW-RSC	Undistortion		CNNs
2022	CVPR	EvUnroll: Neuromorphic Events based Rolling Shutter Image Correction	EvUnroll	Undistortion	PyTorch	U-Net
2022	CVPR	Learning to Detect Scene Landmarks for Camera Localization	Do et al.	Extrinsics	PyTorch	ResNet
2022	CVPR	DiffPoseNet: Direct Differentiable Camera Pose Estimation	DiffPoseNet	Extrinsics	PyTorch	CNNs + LSTM
2022	CVPR	SceneSqueezer: Learning to Compress Scene for Camera Relocalization	SceneSqueezer	Extrinsics	PyTorch	Transformer
2022	arXiv	FishFormer: Annulus Slicing-based Transformer for Fisheye Rectification with Efficacy Domain Exploration	FishFormer	Undistortion	PyTorch	Transformer
2022	CVPR	Focal Length and Object Pose Estimation via Render and Compare	FocalPose	Intrinsics + Extrinsics	PyTorch	CNNs
2022	arXiv	DXQ-Net: Differentiable LiDAR-Camera Extrinsic Calibration Using Quality-aware Flow	DXQ-Net	Camera + LiDAR	PyTorch	CNNs + RNNs
2022	ITSC	SST-Calib: Simultaneous Spatial-Temporal Parameter Calibration between LIDAR and Camera	SST-Calib	Camera + LiDAR	PyTorch	CNNs
2022	IROS	Learning-Based Framework for Camera Calibration with Distortion Correction and High Precision Feature Detection	CCS-Net	Undistortion	PyTorch	UNet
2022	TIP	SIR: Self-supervised image rectification via seeing the same scene from multiple different lenses	SIR	Undistortion	PyTorch	ResNet
2022	TIV	ATOP: An Attention-to-Optimization Approach for Automatic LiDAR-Camera Calibration Via Cross-Modal Object Matching	ATOP	Camera + LiDAR		CNNs
2022	ICRA	FusionNet: Coarse-to-Fine Extrinsic Calibration Network of LiDAR and Camera with Hierarchical Point-pixel Fusion	FusionNet	Camera + LiDAR	PyTorch	CNNs+PointNet
2022	TIM	Keypoint-Based LiDAR-Camera Online Calibration With Robust Geometric Network	RGKCNet	Camera + LiDAR	PyTorch	CNNs+PointNet
2022	ECCV	Rethinking generic camera models for deep single image camera calibration to recover rotation and fisheye distortion	GenCaliNet	Intrinsics + Extrinsics + Distortion coefficients		DenseNet
2022	PAMI	Content-Aware Unsupervised Deep Homography Estimation and Beyond	Liu et al.	Projection matrixs	PyTorch	ResNet

🚩Datasets

Name	Publication	Real/Synthetic	Image/Video	Objectives	Dataset
KITTI	CVPR	Real	Video	Base	Dataset
MS-COCO	ECCV	Real	Image	Base	Dataset
SUN360	CVPR	Real	Image	Base	Dataset
Places2	PAMI	Real	Image	Base	Dataset
CelebA	ICCV	Real	Image	Base	Dataset
1DSfM	ECCV	Real	Image	Focal Length	Dataset
Cambridge Landmarks	ICCV	Real	Video	Extrinsics	Dataset
HLW	BMVC	Real	Image	Horizon Line	Dataset
YUD	ECCV	Real	Image	Vanishing Point	Dataset
ECD	ECCV	Real	Image	Vanishing Point	Dataset
SU3 Wireframe	ICCV	Synthetic	Image	Vanishing Point	Dataset
ScanNet	CVPR	Real	Video	Extrinsics	Dataset
Indoor-6	CVPR	Real	Image	Extrinsics	Dataset
DeepVP	ICRA	Real	Image	Vanishing Point	Dataset
CAHomo	ECCV	Real	Video	Homography	Dataset
MHN	CVPR	Real	Video	Homography	Dataset
UDIS	TIP	Real	Video	Homography	Dataset
Carla-RS	CVPR	Synthetic	Video	RS-Distortion	Dataset
Fastec-RS	CVPR	Synthetic	Video	RS-Distortion	Dataset
BS-RSC	CVPR	Real	Video	RS-Distortion	Dataset
GEV-RS	CVPR	Real	Video	RS-Distortion	Dataset
LMS	ICASSP	Both	Video	Radial Distortion	Dataset
SS-WPC	CVPR	Real	Image	Radial Distortion	Dataset

📜License

The survey and benchmark are only made available for academic research purposes.

📚Citation

@article{kang2023deep,
Author = {Kang Liao and Lang Nie and Shujuan Huang and Chunyu Lin and Jing Zhang and Yao Zhao and Moncef Gabbouj and Dacheng Tao},
Title = {Deep Learning for Camera Calibration and Beyond: A Survey},
Year = {2023},
Journal = {arXiv:2303.10559}
}

🚩知乎导读

导读链接

📭Contact

[e-mail]