Stereo rig with unknown in-camera distortion correction
KevinCain opened this issue · comments
Is there a provision in SS to skip intrinsics calculations for one or both member cameras in a stereo rig? For example -- is there a way to lock an camera with an identity matrix and similarly lock zeros for the distortion coefficients, so that we can be sure that a subsequent SS undistortion process itself is not introducing artifacts or errors?
When using SS to build and rectify a stereo rig where one of the two cameras produces corrected images by an unknown process, I've found it can be destructive to attempt to calibrate the ostensibly distortion-corrected frame. If I had to guess, I would suppose it's because the frames have been cropped to hide the margins after distortion correction, so an attempt to calibrate the cropped images introduces pincushion distortion.
Yes, if calibration is not appropriate most of the times distortion correction worsens final result instead of improving. IMHO, those situations are symptoms of poor calibration.
Anyway, working with a projector it happened to disable the distortion correction, which you can do as simple as:
import os
import simplestereo as ss
# Paths
curPath = os.path.dirname(os.path.realpath(__file__))
loadFile = os.path.join(curPath,"rig.json") # StereoRig file
# Load stereo rig from file
rig = ss.StereoRig.fromFile(loadFile)
rig.distCoeffs1 = None # Remove distortion correction. Internally this is equivalent of having all zeroed coefficients.
rig.distCoeffs2 = None
# Optionally save to file the modified rig
saveFile = os.path.join(curPath,"rig_no_distortion.json") # Destination
rig.save(saveFile)
# or use it straightaway.
However the best thing would be to directly disable the distortion correction while performing calibration.
Since ss is meant for common use cases, the use of the standard 5 coefficients is hard coded in https://github.com/decadenza/SimpleStereo/blob/d104dba8cddd7e43156059d68f2abdba2538a88e/simplestereo/calibration.py#L156C1-L156C1
However, the calibration function ss.calibration.chessboardStereo may be modified to accept different setups, actually I've just found a comment I left at
SimpleStereo/simplestereo/calibration.py
Line 152 in d104dba
So definitely this can be done. I'll plan it for next release.
I implemented the possibility to set 0, 5 or 8 parameters for distortion correction. The advanced options (12 or 14) are not implemented yet.
See latest release 1.0.7 (also via pip install simplestereo)
For your use case, just set distortionCoeffsNumber=0 to disable distortion correction entirely.
Thanks @decadenza for your characteristically fast and helpful effort!
Unless I'm missing something obvious, I believe this uses a single value for the distortion coefficients for both left and right images in the stereo pair. In my case we have two cameras, one that produces distorted ~120^ FOV images and the other which produces undistorted images in-camera through some black-box operation that likely changes the intrinsics, e.g.: by cropping the image to remove black pixels at the perimeter after undistortion.
In this case I changed your current method to allow us to set the number of distortion coefficients for the left and right images in the stereo pair separately, as below. Will this approach have unintended consequences for rig rectification downstream?
stereoRig = chessboardStereo(images, distortionCoeffsNumberLeft=5, distortionCoeffsNumberRight=0)
import numpy as np
import cv2
def chessboardStereo(images, chessboardSize=DEFAULT_CHESSBOARD_SIZE, squareSize=1,
distortionCoeffsNumberLeft=5, distortionCoeffsNumberRight=5):
"""
Performs stereo calibration between two cameras with separate distortion coefficients for each camera.
Parameters
----------
images : list or tuple
A list (or tuple) of 2 dimensional tuples (ordered left and right) of image paths.
chessboardSize: tuple
Chessboard internal dimensions as (width, height).
squareSize : float
The size of a square in your defined world units.
distortionCoeffsNumberLeft: int
The number of distortion coefficients for the left camera.
distortionCoeffsNumberRight: int
The number of distortion coefficients for the right camera.
Returns
-------
StereoRig
A calibrated stereo rig object.
"""
# Arrays to store object points and image points from all the images.
objp = np.zeros((chessboardSize[0]*chessboardSize[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:chessboardSize[0], 0:chessboardSize[1]].T.reshape(-1, 2) * squareSize
imagePoints1 = [] # Image points for the left camera
imagePoints2 = [] # Image points for the right camera
# Process each pair of images
for path1, path2 in images:
img1 = cv2.imread(path1, cv2.IMREAD_GRAYSCALE)
img2 = cv2.imread(path2, cv2.IMREAD_GRAYSCALE)
if img1 is None or img2 is None:
raise ValueError(f"One of the images in {path1}, {path2} not found!")
# Find the chessboard corners
ret1, corners1 = cv2.findChessboardCorners(img1, chessboardSize)
ret2, corners2 = cv2.findChessboardCorners(img2, chessboardSize)
# If found, add object points, image points (after refining them)
if ret1 and ret2:
corners1 = cv2.cornerSubPix(img1, corners1, DEFAULT_CORNERSUBPIX_WINSIZE, (-1,-1), DEFAULT_TERMINATION_CRITERIA)
corners2 = cv2.cornerSubPix(img2, corners2, DEFAULT_CORNERSUBPIX_WINSIZE, (-1,-1), DEFAULT_TERMINATION_CRITERIA)
imagePoints1.append(corners1)
imagePoints2.append(corners2)
# Calibration flags for each camera
flagsLeft = _getCalibrationFlags(distortionCoeffsNumberLeft)
flagsRight = _getCalibrationFlags(distortionCoeffsNumberRight)
flags = flagsLeft | flagsRight # Combine flags for both cameras
# Initialize camera matrices and distortion coefficients
cameraMatrix1 = np.eye(3, dtype=np.float64)
cameraMatrix2 = np.eye(3, dtype=np.float64)
distCoeffs1 = np.zeros(distortionCoeffsNumberLeft)
distCoeffs2 = np.zeros(distortionCoeffsNumberRight)
# Stereo calibration
retval, cameraMatrix1, distCoeffs1, cameraMatrix2, distCoeffs2, R, T, E, F = cv2.stereoCalibrate(
[objp] * len(imagePoints1), imagePoints1, imagePoints2,
cameraMatrix1, distCoeffs1, cameraMatrix2, distCoeffs2,
imageSize=img1.shape[::-1], flags=flags, criteria=DEFAULT_TERMINATION_CRITERIA)
# Create a StereoRig object (you need to define StereoRig according to your needs)
stereoRigObj = StereoRig(imageSize1=img1.shape[::-1][:2],
imageSize2=img2.shape[::-1][:2],
cameraMatrix1=cameraMatrix1,
cameraMatrix2=cameraMatrix2,
distCoeffs1=distCoeffs1,
distCoeffs2=distCoeffs2,
R=R, T=T, E=E, F=F,
reprojectionError=retval)
return stereoRigObj