Thermal/color imagery registration
pbaer opened this issue · comments
We will need the ability to register our thermal and color imagery pairs. Due to shifts in aircraft orientation, etc., image pairs taken at the same time will have some arbitrary offset relative to each other.
Just to add possible ideas / actions items how people can contribute:
- It would be very useful to know, if the offset between images is consistent or not. We might use the annotations provided for that, i.e. the hotspot detections and corresponding bounding box annotations color images.
- If the offset is not consistent, maybe there are subsets of images with consistent offsets. For example, the offset should be consistent for all images of the same flight. A possible task here is to identify these subsets.
- It would be good to have a list of methods, which allow for mapping the IR image to the color image. It might be necessary to select approaches, which also work with few / sparse correspondences. If somebody is interested in image registration, he/she can this list as guidance.
- If somebody is interested in trying out an approach, you could pick one of the list generated in the previous point or any other approach you found useful. Important: please tell others about the outcome, also if it failed. This avoids that we are doing double work :)
If you should decide to work on one of these, it might make sense to create a new issue for each individual task.
Hey @Marcel-Simon, I'd be happy to look into this. For the list of methods, do you have anything in mind already?
I did a quick google search and found the following packages that should work out of the box:
- OpenCV, tutorial, specs
, this has a BSD license, and has interfaces in python & C++ - MATLAB also has something relevant: tutorial, however I'm not sure about licensing here
Also, here are some relevant papers, although they don't seem to be of much practical use: stars-project.org
Hey Matt,
great, thanks! I do not have a list in mind already, so your list is already a great start. If you are interested, you could try some of these approaches. After some discussions earlier, it might make sense to make the IR image more similar to the color images. That means, I could imagine converting the 16-bit IR image into a usable 8 bit image and maybe inverting it, such that the ice is white and the ocean is dark. This should help a lot when registering. However, feel free to try any idea you have in your mind. I am also happy to have a chat anytime.
@Marcel-Simon, the conversion does sound like a good first step; I believe @jomalsan-MSFT is working on this. @jomalsan-MSFT, please let us know how this goes!
@Marcel-Simon and @ms-mkd I am working on this right now. I had a lot of computer issues yesterday, but I have straightened them out and am making progress again. I am targeting this afternoon for having an idea of the best way to normalize the images. Definitely by the beginning of the day tomorrow
Eric started working on the 16-to-8 bit problem yesterday and made some good progress; please sync up with him first. We need to get a solution done this morning as it's blocking both the hotspot detection and the registration workstreams. The first pass of it doesn't have to be perfect :)
