SpectacularAI / HybVIO

Hi, and thank you for making tis code available. I am building it on windows desktop, visual studio 2019, and after a day or so of tweaking, I have it compiled.

However, when I run
main.exe -i=output p -useStereo
The application does not start. ("output" is the dir containing the csv and video files data.csv, data.avi., data2.avi)

I think this is because i am not passing in parameters correctly.

I am confused as to how the parameter json is created.
The system looks for:
std::ifstream cmdParametersFile("../data/cmd.json");

But how do I create this file and pass in the camera intrinsics etc?

(I am using Zed2 data recorded with the zed capture application from your readme)

Thank you!

Hello, you indeed need to include a camera calibration in your data recording folder. To do that, put in your output/ folder a file parameters.txt OR calibration.json. The format of parameters.txt is given here, and an example calibration.json looks like [1]. At the very least you need to include the focal length parameters, principal points, and the two imuToCamera matrices. For better results you also need to specify a camera model and corresponding distortion parameters for both cameras, or otherwise the pinhole model gets used. While you could create the calibration file manually, we also provide some helper code for converting input and output from two calibration tools:

Kalibr: These convert a data folder with data.avi, data2.avi, and data.jsonl to format Kalibr understands and perform the calibration, but you need to manually convert the output into a calibration file that HybVIO understands.
- method 1
- method 2
Basalt: Runs calibration on a data folder with data.avi, data2.avi, and data.jsonl and produces a ready-to-use calibration.json.

The file cmd.json is not required — it can be used for setting defaults for visualizations and such. If you created that file you should probably remove it because it can crash the program if formatted incorrectly. Also in your example line main.exe -i=output p -useStereo you must put a dash before the p parameter: -p.

Thanks for reporting the issue, we will try to improve the documentation regarding this. Please ask if you have further questions.

[1]: calibration.txt The extension has been changed from .json to .txt due to a GitHub upload limitation.

Thank you! I will take a look at this.
One more question, I would like to test this system online, with a zed2 sensor, do you have any examples of running online?

Thanks again!

We do not have any online examples in this repo.

Hi, thank you for the info. i now have a calibration.json file, and running the application with the file gives me the following error:

D:\tracking_2022\HybVIO\build\Release>main.exe -i=D:\\tracking_2022\\HybVIO\\build\\Release\\output
             loguru.cpp:610   arguments: odometry
             loguru.cpp:613   Current dir: D:\tracking_2022\HybVIO\build\Release
             loguru.cpp:615   stderr verbosity: 1
             loguru.cpp:616   -----------------------------------
The system cannot find the path specified.
Failed to extract fps information from video. fpsText:
The system cannot find the path specified.
Failed to extract resolution information from video. resolutionText:
                api.cpp:88    New tracking session, input resolution 600 x 176
                api.cpp:94    Focal length is set to -1 (or 534.486, 534.92), principal point (635.251, 351.015)
                api.cpp:886   Initializing CPU image processing
   feature_detector.cpp:602   initialized feature detector (CPU-fallback)
              image.cpp:354   input is BGR/RGB(A), 1280 x 720. Converting color -> gray
          allocator.hpp:39    increased Allocator capacity 0 -> 5
                api.cpp:927   input -> color frame swizzle: rgb1
          allocator.hpp:39    increased Allocator capacity 0 -> 5
        sample_sync.cpp:310   Camera frame #1 with large time difference to leader: (0.0104)s
          allocator.hpp:39    increased Allocator capacity 0 -> 5
          allocator.hpp:39    increased Allocator capacity 0 -> 5
          allocator.hpp:39    increased Allocator capacity 0 -> 5
          allocator.hpp:39    increased Allocator capacity 0 -> 5
ERROR: assertion rowWidthPixels > 0 failed in accelerated::cpu::Image::createReference (D:\tracking_2022\HybVIO\3rdparty\mobile-cv-suite\accelerated-arrays\src\cpu\image.cpp:220)

My video files are data.avi and data2.avi and do seem to be loaded okay. What might be happening here?

Thanks again.

Sounds like it doesn't find the video file or program ffprobe. What happens if you run ffprobe D:\tracking_2022\HybVIO\build\Release\output\data.avi on command line?

Ah, i get:

'ffprobe' is not recognized as an internal or external command, operable program or batch file.

is that part of ffmpeg? Should the ffprobe.exe be in the same dir as the main.exe file?

Downloaded and added ffprobe.exe. running from the same dir as main.exe gives me:

Input #0, avi, from 'D:\tracking_2022\HybVIO\build\Release\output\data.avi':
  Duration: 00:00:17.55, start: 0.000000, bitrate: 82200 kb/s
  Stream #0:0: Video: mjpeg (Baseline) (MJPG / 0x47504A4D), yuvj420p(pc, bt470bg/unknown/unknown), 1280x720 [SAR 1:1 DAR 16:9], 82264 kb/s, 60 fps, 60 tbr, 60 tbn, 60 tbc

Yes it's part of ffmpeg. It should be on path. You might need to modify this file, so it works on Windows: https://github.com/SpectacularAI/HybVIO/blob/main/src/commandline/videoutil.cpp

Okay i have added ffprobe, and I have validated the frames by viewing with opencv here:

HybVIO/src/commandline/main.cpp

Line 778 in e834243

the frames look good, so the video is loading.
I have added a std::cout for

std::cout << frames[0].t << " " <<  frames[0].intrinsic.focalLengthX << " " <<
                    firstImage.width << " " <<
                    firstImage.height << " " << std::endl;

here:

HybVIO/src/commandline/main.cpp

Line 815 in e834243

So I can see the data is loaded as well.

This gives me:

             loguru.cpp:610   arguments: odometry
             loguru.cpp:613   Current dir: D:\tracking_2022\HybVIO\build\Release
             loguru.cpp:615   stderr verbosity: 1
             loguru.cpp:616   -----------------------------------
                api.cpp:88    New tracking session, input resolution 1270 x 720
                api.cpp:94    Focal length is set to -1 (or 534.486, 534.92), principal point (635.251, 351.015)
1.64079e+09 -1 1280 720
                api.cpp:886   Initializing CPU image processing
   feature_detector.cpp:602   initialized feature detector (CPU-fallback)
              image.cpp:354   input is BGR/RGB(A), 1280 x 720. Converting color -> gray
          allocator.hpp:39    increased Allocator capacity 0 -> 5
                api.cpp:927   input -> color frame swizzle: rgb1
          allocator.hpp:39    increased Allocator capacity 0 -> 5
1.64079e+09 -1 1280 720
        sample_sync.cpp:310   Camera frame #1 with large time difference to leader: (0.0104)s
          allocator.hpp:39    increased Allocator capacity 0 -> 5
          allocator.hpp:39    increased Allocator capacity 0 -> 5
          allocator.hpp:39    increased Allocator capacity 0 -> 5
          allocator.hpp:39    increased Allocator capacity 0 -> 5
1.64079e+09 -1 1280 720
ERROR: assertion rowWidthPixels > 0 failed in accelerated::cpu::Image::createReference (D:\tracking_2022\HybVIO\3rdparty\mobile-cv-suite\accelerated-arrays\src\cpu\image.cpp:220)

It seems to be loading some frames and then failing. Is there anything I can look at to resolve this?

Thanks again for taking the time to help me out!

Parameters?