Evaluation script triggers warnings and ends prematurily with error

[Oliver Kirsebom]

Hello,

First of all - thank you for building and sharing this awesome resource!

Next, a few clarifying questions:

1. As far as I understand, the dataset that you have made publicly available is the 'evaluation dataset' from you arXiv paper (which contains 8,175 frames). Are you also planning on making your training set available too? (I'm particularly interested in the object-detection training set that you've created by merging and re-annotating MODD1, MODD2 and SMD.)
2. Would I be able to run the benchmark on video data that do not have associated IMU data (roll, pitch, yam)?

Finally, a couple of issues I've run into when running the modb_evaluation.py evaluation script.

i. I get a bunch of warnings such as this one

WARN:0@1947.981] global /io/opencv/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('/stru02-00116500-00117000/frames/ignore_mask.png'): can't open/read file: check file path/integrity

(Indeed the ignore_mask.png files appear not to be included in the dataset I've downloaded.)

ii. The script terminates prematurely with the following error message:

Traceback (most recent call last):
  File "/home/oliskir/computer-vision/mods/mods_evaluation/modb_evaluation.py", line 471, in <module>
    run_evaluation()
  File "/home/oliskir/computer-vision/mods/mods_evaluation/modb_evaluation.py", line 324, in run_evaluation
    write_json_file(cfg.PATHS.RESULTS, args.method, evaluation_results)
  File "/home/oliskir/computer-vision/mods/mods_evaluation/utils/utils.py", line 197, in write_json_file
    with open(os.path.join(output_path, output_file_name), 'w') as json_file:
FileNotFoundError: [Errno 2] No such file or directory: '../mods_wasr/results_../wasr_augm0_plainmaster4.json'

Probably something to do with the automatic construction of the output folder path? I've attached my config file to give you an idea of what my directory structure looks like.
mods.txt

Thanks!
Oliver

[bborja]

Hi,

1. Yes, we will also make the training sets for object detection available. I will add a link in the following days.
2. That would require some modifications to the code. Currently, we are using the IMU data to estimate the extent of the "danger zone" in images.

i. Those warnings are nothing to worry about. Ignore masks exist only for sequences where they are needed. If they do not exist for a sequence, that simply means, that there are no parts of the USV visible in the image and that there is no vignette from the lens.
ii. That seems like a problem with paths. Please make sure all paths are specified correctly. The script crashes while trying to access file in directory '../mods_wasr/results_../wasr_augm0_plainmaster4.json'. Please make sure that such folder exists on your machine.

[Oliver Kirsebom]

Thanks for answering my questions, and thanks for making the training sets available too!