Giters

jinlinyi / PerspectiveFields

[CVPR 2023 Highlight] Perspective Fields for Single Image Camera Calibration

Home Page:https://jinlinyi.github.io/PerspectiveFields/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

camera-calibrationcamera-pose-estimation

Perspective Fields for Single Image Camera Calibration

Hugging Face Spaces

Project Page | Paper | Live Demo 🤗

CVPR 2023 (✨Highlight)

Linyi Jin1, Jianming Zhang2, Yannick Hold-Geoffroy2, Oliver Wang2, Kevin Matzen2, Matthew Sticha1, David Fouhey1

1University of Michigan, 2Adobe Research

alt text

We propose Perspective Fields as a representation that models the local perspective properties of an image. Perspective Fields contain per-pixel information about the camera view, parameterized as an up vector and a latitude value.

swiping-1 swiping-2 swiping-3 swiping-4

📷 From Perspective Fields, you can also get camera parameters if you assume certain camera models. We provide models to recover camera roll, pitch, fov and principal point location.

Image 1 Image 2 Image 2

Updates

Table of Contents

Environment Setup

PerspectiveFields requires python >= 3.8 and PyTorch.

| Pro tip: use mamba in place of conda for much faster installs. The dependencies can be installed by running:

git clone git@github.com:jinlinyi/PerspectiveFields.git
# create virtual env
conda create -n perspective python=3.9
conda activate perspective
# install pytorch compatible to your system https://pytorch.org/get-started/previous-versions/
# conda install pytorch torchvision cudatoolkit -c pytorch
conda install pytorch=1.10.0 torchvision torchaudio cudatoolkit=11.3 -c pytorch
# conda packages
conda install -c conda-forge openexr-python openexr
# pip packages
pip install -r requirements.txt
# install mmcv with mim, I encountered some issue with pip install mmcv :(
mim install mmcv
# install Perspective Fields.
pip install -e .

Model Zoo

NOTE: Extract model weights under perspectiveField/models.

Model Name and Weights Training Dataset Config File Outputs Expected input
[NEW]Paramnet-360Cities-edina-centered 360cities and EDINA paramnet_360cities_edina_rpf.yaml Perspective Field + camera parameters (roll, pitch, vfov) Uncropped, indoor🏠, outdoor🏙️, natural🌳, and egocentric👋 data
[NEW]Paramnet-360Cities-edina-uncentered 360cities and EDINA paramnet_360cities_edina_rpfpp.yaml Perspective Field + camera parameters (roll, pitch, vfov, cx, cy) Cropped, indoor🏠, outdoor🏙️, natural🌳, and egocentric👋 data
PersNet-360Cities 360cities cvpr2023.yaml Perspective Field Indoor🏠, outdoor🏙️, and natural🌳 data.
PersNet_paramnet-GSV-centered GSV paramnet_gsv_rpf.yaml Perspective Field + camera parameters (roll, pitch, vfov) Uncropped, street view🏙️ data.
PersNet_Paramnet-GSV-uncentered GSV paramnet_gsv_rpfpp.yaml Perspective Field + camera parameters (roll, pitch, vfov, cx, cy) Cropped, street view🏙️ data.

Coordinate Frame

alt text

yaw / azimuth: camera rotation about the y-axis pitch / elevation: camera rotation about the x-axis roll: camera rotation about the z-axis

Extrinsics: rotz(roll).dot(rotx(elevation)).dot(roty(azimuth))

Inference

python demo/demo.py \
--config-file <config-path> \ #../jupyter-notebooks/models/cvpr2023.yaml 
--input <input-path> \ #../assets/imgs 
--output <output-path> \ #debug 
--opts MODEL.WEIGHTS <ckpt-path> #../jupyter-notebooks/models/cvpr2023.pth

Camera Parameters to Perspective Fields

Checkout Jupyter Notebook. Perspective Fields can be calculated from camera parameters. If you prefer, you can also manually calculate the corresponding Up-vector and Latitude map by following Equations 1 and 2 in our paper. Our code currently supports:

  1. Pinhole model [Hartley and Zisserman 2004] (Perspective Projection)
from perspective2d.utils.panocam import PanoCam
# define parameters
roll = 0
pitch = 20
vfov = 70
width = 640
height = 480
# get Up-vectors.
up = PanoCam.get_up(np.radians(vfov), width, height, np.radians(pitch), np.radians(roll))
# get Latitude.
lati = PanoCam.get_lat(np.radians(vfov), width, height, np.radians(pitch), np.radians(roll))
  1. Unified Spherical Model [Barreto 2006; Mei and Rives 2007] (Distortion).
xi = 0.5 # distortion parameter from Unified Spherical Model

x = -np.sin(np.radians(vfov/2))
z = np.sqrt(1 - x**2)
f_px_effective = -0.5*(width/2)*(xi+z)/x
crop, _, _, _, up, lat, xy_map = PanoCam.crop_distortion(equi_img,
                                             f=f_px_effective,
                                             xi=xi,
                                             H=height,
                                             W=width,
                                             az=yaw, # degrees
                                             el=-pitch,
                                             roll=-roll)

Visualize Perspective Fields

We provide a one-line code to blend Perspective Fields onto input image.

import matplotlib.pyplot as plt
from perspective2d.utils import draw_perspective_fields
# Draw up and lati on img. lati is in radians.
blend = draw_perspective_fields(img, up, lati)
# visualize with matplotlib
plt.imshow(blend)
plt.show()

Perspective Fields can serve as an easy visual check for correctness of the camera parameters.

  • For example, we can visualize the Perspective Fields based on calibration results from this awesome repo.

alt text

  • Left: We plot the perspective fields based on the numbers printed on the image, they look accurate😊;

  • Mid: If we try a number that is 10% off (0.72*0.9=0.648), we see mismatch in Up directions at the top right corner;

  • Right: If distortion is 20% off (0.72*0.8=0.576), the mismatch becomes more obvious.

Training

[Click to expand]

Datasets

We used Google street view dataset to train the ParamNet in our paper Table 3, 4. Download GSV datasets:

wget https://www.dropbox.com/s/plcmcza8vfmmpkm/google_street_view_191210.tar
wget https://www.dropbox.com/s/9se3lrpljd59cod/gsv_test_crop_uniform.tar

Extract the dataset under perspectiveField/datasets.

Note that we used images from 360cities to train the PerspectiveNet in Table 1.

Training PerspectiveNet + ParamNet

Download initial weights from segformer.b3.512x512.ade.160k.pth:

wget https://www.dropbox.com/s/0axxpfga265gq3o/ade_pretrained.pth

Place it under perspectiveField/init_model_weights.

  • We first trained PerspectiveNet:
python -W ignore train.py \
--config-file configs/config-mix-gsv-regress.yaml \
--num-gpus 2 \
--dist-url tcp://127.0.0.1:$((RANDOM +10000)) \
OUTPUT_DIR "./exp/step01-gsv-perspective-pretrain" \
SOLVER.IMS_PER_BATCH 64
  • Then we trained the ParamNet, you can download the model from the previous step here:
wget https://www.dropbox.com/s/c9199n5lmy30tob/gsv_persnet_pretrain.pth
  • To train the ParamNet to predict roll, pitch and fov:
python -W ignore train.py \
--config-file configs/config-gsv-rpf.yaml \
--num-gpus 2 \
--dist-url tcp://127.0.0.1:$((RANDOM +10000)) \
OUTPUT_DIR "./exp/step02-gsv-paramnet-rpf"
  • To train the ParamNet to predict roll, pitch, fov, and principal point:
python -W ignore train.py \
--config-file configs/config-gsv-rpfpp.yaml \
--num-gpus 2 \
--dist-url tcp://127.0.0.1:$((RANDOM +10000)) \
OUTPUT_DIR "./exp/step02-gsv-paramnet-rpfpp"

Testing

[Click to expand]

Datasets

In our paper, we tested the PersNet-360Cities model on images from publicly available datasets Stanford2D3D and TartanAir. Results can be found in Table 1.

To download Stanford2d3d dataset: First agree to their data sharing and usage term: link.

https://www.dropbox.com/sh/ycd4hv0t1nqagub/AACjqZ2emGw7L-aAJ1rmpX4-a

Download TartanAir dataset:

https://www.dropbox.com/sh/7tev8uqnnjfhzhb/AAD9y_d1DCcoZ-AQDEQ1tn0Ua

Extract the datasets under perspectiveField/datasets.

We also tested the PersNet_paramnet-GSV-centered and PersNet_Paramnet-GSV-uncentered models on centered and uncentered images from Google Street View (GSV). Results can be found in Tables 3 and 4.

Download GSV datasets:

wget https://www.dropbox.com/s/plcmcza8vfmmpkm/google_street_view_191210.tar
wget https://www.dropbox.com/s/9se3lrpljd59cod/gsv_test_crop_uniform.tar

Extract the datasets under perspectiveField/datasets.

Testing PerspectiveNet + ParamNet

  • PerspectiveNet:

First, to test PerspectiveNet, provide a dataset name corresponding to a name/path pair from perspective2d/data/datasets/builtin.py. Create and provide an output folder under perspectiveField/exps. Choose a model and provide the path to the config file and weights, both of which should be under perspectiveField/models.

Example:

python -W ignore demo/test_warp.py \
--dataset stanford2d3d_test \
--output ./exps/persnet_360_cities_test \
--config-file ./models/cvpr2023.yaml \
--opts MODEL.WEIGHTS ./models/cvpr2023.pth
  • ParamNet:

To test ParamNet, again provide a dataset name, output folder, and a path to config and model weights, just as with PerspectiveNet.

Example:

python -W ignore demo/test_param_network.py \
--dataset gsv_test  \
--output ./exps/paramnet_gsv_test \
--config-file ./models/paramnet_gsv_rpf.yaml \
--opts MODEL.WEIGHTS ./models/paramnet_gsv_rpf.pth

Citation

If you find this code useful, please consider citing:

@inproceedings{jin2023perspective,
      title={Perspective Fields for Single Image Camera Calibration},
      author={Linyi Jin and Jianming Zhang and Yannick Hold-Geoffroy and Oliver Wang and Kevin Matzen and Matthew Sticha and David F. Fouhey},
      booktitle = {CVPR},
      year={2023}
}

Acknowledgment

This work was partially funded by the DARPA Machine Common Sense Program. We thank authors from A Deep Perceptual Measure for Lens and Camera Calibration for releasing their code on Unified Spherical Model.

About

[CVPR 2023 Highlight] Perspective Fields for Single Image Camera Calibration

https://jinlinyi.github.io/PerspectiveFields/

camera-calibrationcamera-pose-estimation

License:Other

Languages

Language:Jupyter Notebook 90.9%Language:Python 9.1%