Implement stereo camera calibration routines

[Matt Clarkson]

[Stephen Thompson]

Hello Steven,

thanks for your questions, here are some answers. I've copied in Matt and Tom in case they disagree or want to add anything. At the bottom I've put some links to the wiki and slack channel. However we try and keep most of our discussions to the specific issue trackers on Weisslab or now github, that way we can hopefully go back and work out why we did stuff more easily. I've also attached the latest revision of the SNAPPY paper which is in the final stages of revision before IPCAI.

The main thing with SNAPPY is to try and keep each library smallish and orthogonal, i.e. do one thing and minimise dependencies. It's really easy to manage lots of small libraries in Python, so we're trying to take advantage of it. Obviously sometimes libraries may grow too large and need dividing, or vice versa.

So surgeryimage should do image acquisition and some basic processing (feature detection, deinterlacing, undistortion, rectification etc).

surgerycalibration should do the calibration, with various implementations. So yes, I think the calibration bits should be moved out of surgeryimage and into surgerycalibration. Rather than making surgerycalibration a dependency of surgeryimage, the calibration bits of surgeryimage could just be depreciated with suitable error messages telling the developer where to find them now. Updating any dependent applications should then be straightforward.

I'm not sure if surgeryimage should be a dependency of surgerycalibration, to provide some image acquisition and basic processing.

There isn't really a set data flow for SNAPPY, as that would be application specific. SNAPPY just provides the libraries, which can then be assembled as the application developer wishes.

Adding package specific data types has pros and cons as you point out. We want the libraries to be as reusable as possible, so ideally should be avoided. One option is to put them into surgerycore, which all SNAPPY libraries can depend on. I have just added an abstract base class for scikitsurgery-trackers to it, so if you think it will be useful, you could do something similar for calibration data types.

thanks and regards,

Steve.

On 06/04/2020 16:08, Hirschmann, Steven wrote:

Hello Stephen,

I am looking at doing some work on the SNAPPY platform for the calibration functionality.

I was informed that the work will be going into the scikit-surgerycalibration repository, and I had some questions about systematically how you would prefer I go about this.

I am interested in how you imagine process flow and dataflow between the different packages. Primarily between surgerycalibration and surgeryimage.

currently surgeryimage has about 90% of what we would want in surgerycalibration, we just need to expand it to include the implementations of mono and stereo calibrations from checkerboard and aruco boards.

would you be happy to move those calibration and detection elements to surgerycalibration, and then make it a dependency of surgeryimage?

The alternative would be making surgerycalibration dependent on surgeryimage which seems counterintuitive.

is there documentation regarding the envisioned distribution of responsibilities and dataflow for the platform?
IE: you intend that if you want to do aruco tracking, there is a data flow of:
surgerycalibration (to generate intrinics) -> surgeryarucotracking (to estimate the pose) -> surveryvtk (to generate an output image).

Would you prefer we avoided created package specific data types (IE: a calibration type might have a camera matrix, distortion coefficients, object points, image dimensions, rotation and translation vectors).

Creating them makes keeping track of the data easier by organizing it relationally, but also opens a can of worms architecturally to avoid mandated dependency.

Lastly, is there a slack/teams/email chain where architecture is discussed.

Sorry for the many questions, but I’m trying to understand your design philosophy so that I can properly design my code to conform.

Kind regards,
Steven hirschmann

[Stephen Thompson]

I agree with this. I think that scikit-surgerycalibration should use image acquisition/processing and feature detectors implemented in scikit-surgeryimage. So yes, scikit-surgeryimage will be a dependency.

It will also be good if there are some calibration apps as part of the library so you can run it. As long as we're disciplined with keeping the algorithms and the user interface separate this shouldn't be a problem. So calibration algorithms go in sksurgerycalibration/algorithms or similar. User interfaces go in sksurgerycalibration/ui or similar. If in the future we think we need to split the library to manage dependencies, then that should be fairly straightforward.

regards,

Steve.
On 08/04/2020 09:52, Clarkson, Matt wrote:

Hi All,

I made some progress yesterday. I can’t see Steve, Francisco or Dimitris on SNAPPY Slack channel, so I’m sticking to email for now.

My email below basically said that each “customer” of snappy should maintain their own application level python package, like SmartLiver does. This is probably good from an architecture point of view, and it gives maximum flexibility to develop the SNAPPY libraries independently. BUT, if we consider the usability of the end result, I came to a different conclusion. For example, someone who “just wants to calibrate” won’t want to create their own standalone project. They would want to work in scikit-surgerycalibration. Also, further down the line, scikit-surgerycalibration would become a publishable entity, and that would be easier if there were standalone, runnable apps therein. So, if we consider that these benefits outweigh the “small, independent library” philosophy, then I think it would be advantageous to ensure that command line apps at least are runnable from scikit-surgerycalibration. If that was the case I would also make scikit-surgeryimage a dependency, as video calibration will always need images, and feature detectors.

What do we think?

M

[tdowrick]

@MattClarkson

Moving the conversation here from weisslab.

I have corrected the matrix multiplication ordering (I think - e4f31f7) and now everything seems quite sensible. Getting ~13 for the reprojection error and ~1.3 for reconstruction error using handeye calibration on the test data set.

For comparision, the reproj/recon error after stereo calibration are ~0.6 and ~1.6 respectively.

[tdowrick]

Right, I think we're getting there.

I've done some more general refactoring, and added in code so that MonoVideoCalibrationDriver can do a mono handeye calibration.

Mono reprojection/reconstruction errors coming out as 14.94/5.53 on the test data using the left channel as input.