How to build cv2 in debug non-optimized mode?

Question

How to build cv2 in debug non-optimized mode?

ApproximateIdentity opened this issue 3 years ago · comments

debian 10
gcc/g++ version 8.3.0 (Debian 8.3.0-6)
Python 3.9.0

I want to build the final cv2 shared library (e.g. cv2.cpython-39-x86_64-linux-gnu.so) with debug symbols and without optimizations. I have tried various strategies like editing the opencv/CMakeLists.txt as well as running a build with something like this:

export ENABLE_HEADLESS=1
export CMAKE_ARGS='-DCMAKE_BUILD_TYPE=DEBUG -DCMAKE_C_FLAGS_DEBUG="-g -O0" -DCMAKE_CXX_FLAGS_DEBUG="-g -O0"'
TMPDIR=$PWD/build pip wheel --no-clean --verbose .

No matter what I do, the final library seems to be built in Release mode and ignores my optimizations. How do I do this?

Thomas Nyberg · Answer 1 · Thu Dec 03 2020 20:48:13 GMT+0800 (China Standard Time)

Alright I came up with an extremely hacky solution that does work. Someone more used to cmake would probably laugh at this solution, but it does work (and is faster than my trying to fight a build system I don't understand). I'll leave it here for the record so that I can move on, but I'm sure someone here can pop in and explain the right way to do this.

I ran the build as follows choosing to keep all temporary build files:

build.sh

export ENABLE_HEADLESS=1
export VERBOSE=1

mkdir build
TMPDIR=$PWD/build pip wheel --no-clean --verbose .

Then I swallowed up the output into build.log:

$ bash my_build.sh 2>&1 | tee build.log

Then I extracted and transformed all commands involving the build:

hack_build.py

#!/usr/bin/env python3

import sys

for line in sys.stdin:
    if (line.startswith('  cd') or
        line.startswith('  /usr/bin/ar') or
        line.startswith('  /usr/bin/c++') or
        line.startswith('  /usr/bin/ranlib')):
        newline = line.strip(' ').replace('-O3', '-O0 -g')
        sys.stdout.write(newline)

$ python3 hack_build.py < build.log > hacked_build.sh

Finally executing $ bash hacked_build.sh re-built the shared library without any optimizations. On my computer it produced the file:

./build/pip-req-build-ro720y4z/_skbuild/linux-x86_64-3.9/cmake-build/lib/python3/cv2.cpython-39-x86_64-linux-gnu.so

After copying that file into the directory path/to/lib/python3.9/site-packages/cv2/ I replaced the other version with mine and it works.

Olli-Pekka Heinisuo · Answer 2 · Fri Dec 04 2020 00:29:23 GMT+0800 (China Standard Time)

I believe scikit-build (CMake wrapper library) overwrites the -DCMAKE_BUILD_TYPE=Debug flag here: https://github.com/scikit-build/scikit-build/blob/c0741f9a1de9dd97e0a1e6f0b3b02fcd76221418/skbuild/setuptools_wrap.py#L105.

Because of the pip wheel command, the parameters to scikit-built cannot be passed directly as command line parameters as documented here (due to this there is the CMAKE_ARGS variable in this repository): https://scikit-build.readthedocs.io/en/latest/usage.html?highlight=debug#usage-scikit-build-options

However, you could try running:

pip wheel . --verbose --build-option="--build-type=Debug" (or --global-option, I did not test either one)

but then pip builds the whole dependency chain from source which is slow and not needed: pypa/pip#2677

pip wheel command docs: https://pip.pypa.io/en/stable/reference/pip_wheel/#customising-the-build

OpenCV build information: https://docs.opencv.org/master/db/d05/tutorial_config_reference.html

Thomas Nyberg · Answer 3 · Fri Dec 04 2020 21:13:07 GMT+0800 (China Standard Time)

Thanks for the help, but when I tried pip wheel . --verbose --build-option="--build-type=Debug" I got the following error:

37361 Building wheels for collected packages: opencv-python-headless, numpy
37362   Created temporary directory: /home/user/src/opencv-python/build/pip-wheel-nljd2_ld
37363   ERROR: Cannot build wheel for opencv-python-headless using PEP 517 when --build-option is present
37364   Created temporary directory: /home/user/src/opencv-python/build/pip-wheel-mj3bd_7i
37365   ERROR: Cannot build wheel for numpy using PEP 517 when --build-option is present
37366 Failed to build opencv-python-headless numpy
37367 ERROR: Failed to build one or more wheels
37368 Exception information:
37369 Traceback (most recent call last):
37370   File "/home/user/folder/repos/vision/sandbox/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 210, in _main
37371     status = self.run(options, args)
37372   File "/home/user/folder/repos/vision/sandbox/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
37373     return func(self, options, args)
37374   File "/home/user/folder/repos/vision/sandbox/lib/python3.9/site-packages/pip/_internal/commands/wheel.py", line 184, in run
37375     raise CommandError(
37376 pip._internal.exceptions.CommandError: Failed to build one or more wheels

Given that I only want to do this for investigative/understanding work, my hack seems like the easiest approach for the moment since it already works. I'm guessing there aren't that many people who find themselves in my boat.

But thanks for the response!

Olli-Pekka Heinisuo · Answer 4 · Sat Dec 05 2020 22:06:02 GMT+0800 (China Standard Time)

There's one additional way of doing this via the python setup.py bdist_wheel command. I tested this on Windows and it works:

python setup.py bdist_wheel --build-type=Debug

Please note that you must install at least scikit-build and numpy before that command. Might need also something else, since the tooling does not get installed automatically via pyproject.toml. Due to that, I recommend running the build in a virtual environment which has all needed dependencies installed.

Thomas Nyberg · Answer 5 · Thu Dec 10 2020 06:07:03 GMT+0800 (China Standard Time)

I just checked and this worked for me on Linux too. Might be worth adding something in the main readme about this (which I can try to remember to do), but other than that seems good. Thanks!

Black13 · Answer 6 · Fri Jan 15 2021 01:25:29 GMT+0800 (China Standard Time)

C:\temp\opencv-python>python setup.py bdist_wheel --build-type=Debug
C:\Program Files\Python39\lib\site-packages\setuptools\distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
warnings.warn(
[1/2] Linking CXX shared module lib\python3\cv2.cp39-win_amd64.pyd
FAILED: lib/python3/cv2.cp39-win_amd64.pyd
cmd.exe /C "cd . && "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -E vs_link_dll --intdir=modules\python3\CMakeFiles\opencv_python3.dir --rc=C:\PROGRA~~2\WI3CF2~~1\10\bin\100183~~1.0\x64\rc.exe --mt=C:\PROGRA~~2\WI3CF2~~1\10\bin\100183~~1.0\x64\mt.exe --manifests -- C:\PROGRA~~2\MICROS~~2\2019\COMMUN~~1\VC\Tools\MSVC\1428~~1.293\bin\Hostx64\x64\link.exe /nologo modules\python3\CMakeFiles\opencv_python3.dir_\src2\cv2.cpp.obj /out:lib\python3\cv2.cp39-win_amd64.pyd /implib:lib\opencv_python3.lib /pdb:lib\python3\cv2.pdb /dll /version:0.0 /machine:x64 /debug /INCREMENTAL /NODEFAULTLIB:atlthunk.lib /NODEFAULTLIB:atlsd.lib /DEBUG "C:\Program Files\Python39\libs\python39.lib" lib\opencv_core451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_ml451d.lib lib\opencv_photo451d.lib lib\opencv_dnn451d.lib lib\opencv_features2d451d.lib lib\opencv_imgcodecs451d.lib lib\opencv_videoio451d.lib lib\opencv_calib3d451d.lib lib\opencv_highgui451d.lib lib\opencv_objdetect451d.lib lib\opencv_stitching451d.lib lib\opencv_video451d.lib lib\opencv_gapi451d.lib lib\opencv_videoio451d.lib lib\opencv_imgcodecs451d.lib 3rdparty\lib\libjpeg-turbod.lib 3rdparty\lib\libwebpd.lib 3rdparty\lib\libpngd.lib 3rdparty\lib\libtiffd.lib 3rdparty\lib\libopenjp2d.lib 3rdparty\lib\IlmImfd.lib comctl32.lib gdi32.lib ole32.lib setupapi.lib 3rdparty\lib\quircd.lib lib\opencv_video451d.lib lib\opencv_dnn451d.lib 3rdparty\lib\libprotobufd.lib lib\opencv_calib3d451d.lib lib\opencv_features2d451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_core451d.lib 3rdparty\lib\zlibd.lib 3rdparty\lib\ittnotifyd.lib 3rdparty\lib\ippiwd.lib 3rdparty\ippicv\ippicv_win\icv\lib\intel64\ippicvmt.lib lib\ade.lib wsock32.lib ws2_32.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK Pass 1: command "C:\PROGRA~~2\MICROS~~2\2019\COMMUN~~1\VC\Tools\MSVC\1428~~1.293\bin\Hostx64\x64\link.exe /nologo modules\python3\CMakeFiles\opencv_python3.dir_\src2\cv2.cpp.obj /out:lib\python3\cv2.cp39-win_amd64.pyd /implib:lib\opencv_python3.lib /pdb:lib\python3\cv2.pdb /dll /version:0.0 /machine:x64 /debug /INCREMENTAL /NODEFAULTLIB:atlthunk.lib /NODEFAULTLIB:atlsd.lib /DEBUG C:\Program Files\Python39\libs\python39.lib lib\opencv_core451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_ml451d.lib lib\opencv_photo451d.lib lib\opencv_dnn451d.lib lib\opencv_features2d451d.lib lib\opencv_imgcodecs451d.lib lib\opencv_videoio451d.lib lib\opencv_calib3d451d.lib lib\opencv_highgui451d.lib lib\opencv_objdetect451d.lib lib\opencv_stitching451d.lib lib\opencv_video451d.lib lib\opencv_gapi451d.lib lib\opencv_videoio451d.lib lib\opencv_imgcodecs451d.lib 3rdparty\lib\libjpeg-turbod.lib 3rdparty\lib\libwebpd.lib 3rdparty\lib\libpngd.lib 3rdparty\lib\libtiffd.lib 3rdparty\lib\libopenjp2d.lib 3rdparty\lib\IlmImfd.lib comctl32.lib gdi32.lib ole32.lib setupapi.lib 3rdparty\lib\quircd.lib lib\opencv_video451d.lib lib\opencv_dnn451d.lib 3rdparty\lib\libprotobufd.lib lib\opencv_calib3d451d.lib lib\opencv_features2d451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_core451d.lib 3rdparty\lib\zlibd.lib 3rdparty\lib\ittnotifyd.lib 3rdparty\lib\ippiwd.lib 3rdparty\ippicv\ippicv_win\icv\lib\intel64\ippicvmt.lib lib\ade.lib wsock32.lib ws2_32.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:modules\python3\CMakeFiles\opencv_python3.dir/intermediate.manifest modules\python3\CMakeFiles\opencv_python3.dir/manifest.res" failed (exit code 1104) with the following output:
LINK : fatal error LNK1104: cannot open file 'python39_d.lib'
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "C:\Users\jjosburn\AppData\Roaming\Python\Python39\site-packages\skbuild\setuptools_wrap.py", line 589, in setup
cmkr.make(make_args, env=env)
File "C:\Users\jjosburn\AppData\Roaming\Python\Python39\site-packages\skbuild\cmaker.py", line 496, in make
raise SKBuildError(

An error occurred while building with CMake.
Command:
cmake --build . --target install --config Debug --
Source directory:
C:\temp\opencv-python
Working directory:
C:\temp\opencv-python_skbuild\win-amd64-3.9\cmake-build
Please see CMake's output for more information.

Maik Riechert · Answer 7 · Thu Mar 18 2021 17:41:29 GMT+0800 (China Standard Time)

Having the same issue as @black13. I just need the debug output from https://github.com/opencv/opencv/blob/1363496c1106606684d40447f5d1149b2c66a9f8/modules/videoio/src/cap_dshow.cpp#L302 but looks like to get this I need to get a debug build of python including debug builds of opencv-python dependencies. Is this true?

Thomas Nyberg · Answer 8 · Thu Mar 18 2021 17:50:00 GMT+0800 (China Standard Time)

You should really only need a debug build of the library you're interested in, but it can be a bit tricky. The method I used in my post should work for Linux, but if you're on Windows like black13, I can't really help you with the details.

Olli-Pekka Heinisuo · Answer 9 · Thu Mar 18 2021 18:24:21 GMT+0800 (China Standard Time)

@letmaik @black13 The error you are seeing is caused by the missing Python debug binaries. We can see clearly from the output that the linker is trying to link against the Python debug binary (note the _d):

LINK : fatal error LNK1104: cannot open file 'python39_d.lib'

See for example this issue: opencv/opencv#7395

You must install the debug binaries. This can be done via the official Python installer as documented here (customize installation): https://docs.python.org/3/using/windows.html#installation-steps

Selecting “Customize installation” will allow you to select the features to install, the installation location and other options or post-install actions. To install debugging symbols or binaries, you will need to use this option.

Edit: You can run the installer also without UI if needed: https://docs.python.org/3/using/windows.html#installing-without-ui

Maik Riechert · Answer 10 · Fri Mar 19 2021 06:27:48 GMT+0800 (China Standard Time)

@skvark Thanks for the pointer. I tried to do that here by adding Include_debug=1: https://ci.appveyor.com/project/letmaik/opencv-python/builds/38298354 But I'm getting the same error (see end of log). ~~It seems _d.dll files are installed but no .lib files.~~
EDIT: The .lib files are correctly installed, e.g. python39_d.lib. I think the issue is in the CMake config of opencv somehow: -DPYTHON3_LIBRARY=C:/mypython/libs/python39.lib this should be the debug library. I'm not sure if this is really the root issue however.

Olli-Pekka Heinisuo · Answer 11 · Fri Mar 19 2021 14:47:22 GMT+0800 (China Standard Time)

Scikit-build picks up the wrong path, you can change it here: https://github.com/opencv/opencv-python/blob/master/setup.py#L41

Maik Riechert · Answer 12 · Sat Mar 27 2021 23:40:25 GMT+0800 (China Standard Time)

Scikit-build picks up the wrong path, you can change it here: https://github.com/opencv/opencv-python/blob/master/setup.py#L41

I hacked in the right path (upstream issue: scikit-build/scikit-build#533) and got a bit closer. It now fails during copying of the files from the CMake install folder:

Traceback (most recent call last):
  File "C:\projects\opencv-python\setup.py", line 458, in <module>
    main()
  File "C:\projects\opencv-python\setup.py", line 218, in main
    skbuild.setup(
  File "C:\mypython\lib\site-packages\skbuild\setuptools_wrap.py", line 621, in setup
    _classify_installed_files(cmake_manifest, package_data, package_prefixes,
  File "C:\projects\opencv-python\setup.py", line 371, in _classify_installed_files_override
    raise Exception("Not found: '%s'" % relpath_re)
Exception: Not found: 'python/cv2[^/]*\.cp39\-win_amd64\.pyd'

Here's the log: https://ci.appveyor.com/project/letmaik/opencv-python/builds/38432602

What's weird is that the "-- Installing:" line from CMake for the .pyd is missing, compared to a regular build like https://ci.appveyor.com/project/skvark/opencv-python/builds/36078624/job/vd96v0pmj7ai9bno?fullLog=true.

EDIT: Looks like it's related to this code: https://github.com/opencv/opencv/blob/fc1a15626226609babd128e043cf7c4e32f567ca/modules/python/common.cmake#L125-L129

Deleted user · Answer 13 · Thu Apr 08 2021 02:56:22 GMT+0800 (China Standard Time)

Scikit-build picks up the wrong path, you can change it here: https://github.com/opencv/opencv-python/blob/master/setup.py#L41

I hacked in the right path (upstream issue: scikit-build/scikit-build#533) and got a bit closer. It now fails during copying of the files from the CMake install folder:
Traceback (most recent call last):
  File "C:\projects\opencv-python\setup.py", line 458, in <module>
    main()
  File "C:\projects\opencv-python\setup.py", line 218, in main
    skbuild.setup(
  File "C:\mypython\lib\site-packages\skbuild\setuptools_wrap.py", line 621, in setup
    _classify_installed_files(cmake_manifest, package_data, package_prefixes,
  File "C:\projects\opencv-python\setup.py", line 371, in _classify_installed_files_override
    raise Exception("Not found: '%s'" % relpath_re)
Exception: Not found: 'python/cv2[^/]*\.cp39\-win_amd64\.pyd'
Here's the log: https://ci.appveyor.com/project/letmaik/opencv-python/builds/38432602

What's weird is that the "-- Installing:" line from CMake for the .pyd is missing, compared to a regular build like https://ci.appveyor.com/project/skvark/opencv-python/builds/36078624/job/vd96v0pmj7ai9bno?fullLog=true.

EDIT: Looks like it's related to this code: https://github.com/opencv/opencv/blob/fc1a15626226609babd128e043cf7c4e32f567ca/modules/python/common.cmake#L125-L129

I faced the exact same issue when building wheel on linux.
Exception: Not found: 'python/cv2[^/]*.cpython-36m-x86_64-linux-gnu.so'
did you overcome this eventually?

David V. Corbin · Answer 14 · Sat May 08 2021 21:15:46 GMT+0800 (China Standard Time)

For those who just want a "python39_d.lib" (DLL) what is the option? [No desire to actually BUILD Python]

Boooke · Answer 15 · Sat Apr 09 2022 01:44:51 GMT+0800 (China Standard Time)

Edit: I am working with latest commit on the opencv repo and with the tag 4.5.5

I installed a python 3.10 version with the corresponding python310_d.lib, and also attempted to create a python debug build, though it seemed to be a bigger hassle than it was worth. I couldn't get the installer debug libs to work even when opencv found the python310_d.lib, perhaps due to the problem described above with scikit?

Anyways, prior to this I did something like:
https://stackoverflow.com/questions/50422351/compile-and-link-opencv-to-my-project-in-my-main-cmakelists-txt
and it has worked using two sample programs, and my own so far, so I'll stick to that.
I keep opencv as a git submodule in my project tree as ./third_party/opencv.

In case the stackoverflow topic is lost, this is the minimum CMakeLists.txt for the Hough Transform example:

# Example using the hough transform sample code
project(MyProject)

add_subdirectory(third_party) 
    # which then does add_subdirectory(opencv)

add_executable(MyExe 
    MyHoughTransform.cpp
)

target_link_libraries(MyExe
PRIVATE 
    opencv_core
    opencv_imgcodecs
    opencv_highgui
    opencv_imgproc
)

target_include_directories(MyExe 
PRIVATE 
    third_party/opencv/modules/calib3d/include
    third_party/opencv/modules/core/include
    third_party/opencv/modules/cudaarithm/include
    third_party/opencv/modules/cudabgsegm/include
    third_party/opencv/modules/cudacodec/include
    third_party/opencv/modules/cudafeatures2d/include
    third_party/opencv/modules/cudafilters/include
    third_party/opencv/modules/cudaimgproc/include
    third_party/opencv/modules/cudalegacy/include
    third_party/opencv/modules/cudaobjdetect/include
    third_party/opencv/modules/cudaoptflow/include
    third_party/opencv/modules/cudastereo/include
    third_party/opencv/modules/cudawarping/include
    third_party/opencv/modules/cudev/include
    third_party/opencv/modules/dnn/include
    third_party/opencv/modules/features2d/include
    third_party/opencv/modules/flann/include
    third_party/opencv/modules/highgui/include
    third_party/opencv/modules/imgcodecs/include
    third_party/opencv/modules/imgproc/include
    third_party/opencv/modules/ml/include
    third_party/opencv/modules/objdetect/include
    third_party/opencv/modules/photo/include
    third_party/opencv/modules/shape/include
    third_party/opencv/modules/stitching/include
    third_party/opencv/modules/superres/include
    third_party/opencv/modules/ts/include
    third_party/opencv/modules/video/include
    third_party/opencv/modules/videoio/include
    third_party/opencv/modules/videostab/include
    third_party/opencv/modules/viz/include
    third_party/opencv/modules/world/include
    build
)

The only downside I see is that in addition to the target includes in above link, I have to include my ./build folder since ./build/opencv2/opencv_modules.hpp is referred to in the opencv code inside modules/core/include/opencv2/core/base.hpp.

I have no concrete idea why it really works, I get the same errors and warnings as when I set up for building opencv inside Visual Studio, but the fact that it does work in both Debug and Release is more than enough for me at the moment.