Aborted error in ceres::Solve()

[GoroYeh-HRI]

Hi,
I got the following error when running ceres::Solve()
I've built the executables with -DCMAKE_BUILD_TYPE=Debug
However, I still couldn't tell where this error came from.
Could you guide me on how to debug this error?

*** Aborted at 1706229957 (unix time) try "date -d @1706229957" if you are using GNU date ***
PC: @ 0x555997e3c900 ZZN5ceres8internal14ParallelInvokeIZNS0_15SchurEliminatorILi2ELi3ELin1EE9EliminateERKNS0_21BlockSparseMatrixDataEPKdS8_PNS0_23BlockRandomAccessMatrixEPdEUliE_EEvPNS0_11ContextImplEiiiOT_iENKUlRSF_E_clIKSI_EEDaSH
*** SIGSEGV (@0x7f026be33250) received by PID 3260797 (TID 0x7efee07fc000) from PID 1810051664; stack trace: ***
@ 0x7eff25839631 (unknown)
@ 0x7eff25975420 (unknown)
@ 0x555997e3c900 ZZN5ceres8internal14ParallelInvokeIZNS0_15SchurEliminatorILi2ELi3ELin1EE9EliminateERKNS0_21BlockSparseMatrixDataEPKdS8_PNS0_23BlockRandomAccessMatrixEPdEUliE_EEvPNS0_11ContextImplEiiiOT_iENKUlRSF_E_clIKSI_EEDaSH
@ 0x555997d4a265 ceres::internal::ThreadPool::ThreadMainLoop()
@ 0x7eff240f1df4 (unknown)
@ 0x7eff25969609 start_thread
@ 0x7eff23d9a353 clone
Segmentation fault (core dumped)

[Sameer Agarwal]

it is hard to debug ths with just ths info, is it possible to replicate this failure?
can you tell us more about your build/run environment?
@DmitriyKorchemkin any idea what maybe going on here?

[GoroYeh-HRI]

Hi @sandwichmaker ,
Thank you for the reply!
Yes, the build environment is just COLMAP build environment:
colmap installation

I installed ceres-solver 2.2.0 and verified the version by running simple_bundle_adjustment test.

It aborted during line 327

When building this problem_, we have 60006 images and about 13 millions points.
We want to jointly optimize the image poses and map points using ceres::solve() with :

1. solver_options.linear_solver_type = ceres::DENSE_SCHUR
2. solver_options.dense_linear_algebra_library_type= ceres::DenseLinearAlgebraLibraryType::CUDA;

However, we got this aborted error every time.

If we change the linear_solver_type to ceres::ITERATIVE_SCHUR or ceres::SPARSE_SCHUR it won't abort and can finish the process.

[Sameer Agarwal]

I suspect that dense_schur is not suitable for 60k images, the resulting matrix is too likely too large for your gpu/memory.

…

On Tue, Jan 30, 2024 at 1:56 PM GoroYeh-HRI ***@***.***> wrote: Hi @sandwichmaker <https://github.com/sandwichmaker> , Thank you for the reply! Yes, the build environment is just COLMAP build environment: colmap installation <https://colmap.github.io/install.html#linux> I installed ceres-solver 2.2.0 and verified the version by running simple_bundle_adjustment test. It aborted during line 327 image.png (view on web) <https://github.com/ceres-solver/ceres-solver/assets/137837175/5edac99e-718c-4ba3-808b-6d64966bfd3b> When building this problem_, we have 60006 images and about 13 millions points. We want to jointly optimize the image poses and map points using ceres::solve() with : 1. solver_options.linear_solver_type = ceres::DENSE_SCHUR 2. solver_options.dense_linear_algebra_library_type= ceres::DenseLinearAlgebraLibraryType::CUDA; However, we got this aborted error every time. — Reply to this email directly, view it on GitHub <#1046 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANCABPPVPIYAUL52ZUVUW3YRFUBHAVCNFSM6AAAAABCLOB7G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJXHE3DEMBZGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[GoroYeh-HRI]

@sandwichmaker I see. Thanks for the reply.
After I changed the solver to ceres::ITERATIVE_SCHUR it won't abort anymore.

Since we want to leverage CUDA acceleration and that only supports either DENSE_SCHUR or DENSE_QR solvers,
is there any way we can do it while avoiding aborted error?
Thank you!

[Sameer Agarwal]

Currently there is no way to use CUDA for such large problems.