Crashing during undistortion step
elunty opened this issue · comments
How did you install ODM? (Docker, installer, natively, ...)?
Windows Installer (v3.3.4)
What is the problem?
Program silently crashes during undistortion of images
What should be the expected behavior? If this is a feature request, please describe in detail the changes you think should be made to the code, citing files and lines where changes should be made, if possible.
Not crashing would be good, but giving more logs explaining why it happened would be useful
Log before it "crashes":
[INFO] Undistorting F:\Testing\opendronemap\sample #1\opensfm ...
2023-12-20 09:21:54,986 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000368.jpg
2023-12-20 09:21:54,987 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000386.jpg
2023-12-20 09:21:54,987 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000378.jpg
2023-12-20 09:21:54,988 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000369.jpg
2023-12-20 09:21:54,988 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000387.jpg
2023-12-20 09:21:54,987 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000391.jpg
2023-12-20 09:21:54,989 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000379.jpg
2023-12-20 09:21:54,988 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000390.jpg
2023-12-20 09:21:54,987 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000392.jpg
2023-12-20 09:21:54,988 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000389.jpg
2023-12-20 09:21:54,989 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000393.jpg
2023-12-20 09:21:54,988 DEBUG: Undistorting image panoramic_2023-10-09_16-10-37_000367.jpg
(venv) C:\ODM>
(venv) C:\ODM>
`
How can we reproduce this? What steps did you do to trigger the problem? If this is an issue with processing a dataset, YOU MUST include a copy of your dataset AND task output log, uploaded on Google Drive or Dropbox (otherwise we cannot reproduce this).
Sorry, im not allowed to share the dataset. Is there not anywhere i can look for extra logs about why it is crashing?
Example dataset here:
https://drive.google.com/file/d/1Gr424P_ZaVM50iK3Ydicu2SZr9xgjLwz/view?usp=sharing
Ongoing discussion here:
https://community.opendronemap.org/t/processing-crashes-while-undistorting-panoramic-images-with-no-extra-logs/18683/
In short, it appears that with 8K 360 data, undistort becomes the memory bottleneck rather than our usual texturing step.
@elunty -- I don't know if you want to attempt a patch, but just in case, I'll re-open the issue now that we have an example dataset.
Now testing on Linux with a node that is limited to 32GB to see if we can replicate across operating systems.
I get crashes at different places each time with 32 and 64GB, but I'm not sure how your virtual memory is configured on Windows, so no extra insights from my testing.