Giters

naver / kapture

kapture is a file format as well as a set of tools for manipulating datasets, and in particular Visual Localization and Structure from Motion data.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Multiple types of keypoints and descriptors

ducha-aiki opened this issue · comments

commented

Hi,

I am trying to fit the kapture for my use case, which requires multiple types of local features, both detectors and descriptors, e.g. SuperPoint, R2D2, SIFT+HardNet.
The current kapture format assumes a single type of local feature -- which is hardcoded in the filenames "keypoints.txt", "descriptors.txt". If I would make a fork for support multiple feature types, would it be merged at some point, or do you not plan to change the format?

Best, Dmytro.

commented

Hi Dmytro!

Thanks for your interest in this. Indeed, we already discussed a lot how to support multiple types of local features. We will continue the discussion and get back to you as soon as possible.

Best,
Martin

commented

Thank you, Martin! If possible, I would be happy to join the discussion :)
My email is ducha.aiki@gmail.com just in case

commented

Great! It would also be helpful if you could share some more details about your needs. At the moment we use kapture with multiple features, but not within the same reconstruction.
Best, Martin

commented

Sure. The general idea is to revisit MODS, which combines the multiple type of features in order to speed-up matching of the easy pairs and to allow matching the hard pairs, where any single feature fails, but combined they work.
So, first we are matching everything which is matchable by ORB, then we add, say, SuperPoint, and so on.

https://github.com/ducha-aiki/mods-light-zmq
https://arxiv.org/abs/1503.02619

commented

Hi,
We've started to draft a new version of the format (1.1) that should hopefully answer your needs.
The new specifications are being written here
https://github.com/naver/kapture/blob/1.1_draft/kapture_format.adoc

There are no changes to the "sensor" part of the format.
In the "reconstruction" part, multiple keypoints/descriptors/global features types can be given. keypoints can be represented by multiple descriptors.
Observations can reference different types of keypoints.

What do you think of these changes ?

commented

Hi,

It fits my use-case, thank you!
One thing, I would like to add is a metric for descriptors. So the descriptors.txt would be:

name, dtype, dsize, keypoints_type, metric_type

where the metric is "L2" for the most and "Hamming" for the binary features, for example.