Bolt Python wrapper doesn't build on some systems

Question

Bolt Python wrapper doesn't build on some systems

dblalock opened this issue 7 years ago · comments

At present, Bolt's python code doesn't appear to be building on Ubuntu and hasn't been tested on Windows. It may also have issues building on other systems.

Armin Bashizade · Answer 1 · Sun Jan 07 2018 07:58:11 GMT+0800 (China Standard Time)

I could build the code on redhat with this modifications:

include stdint.h in cpp/src/quantize/bolt.hpp
use swig 3.0
compile with clang 3.9 (in setup.py: extra_args = ['std=c++1y', '-stdlib=libc++', ...] and os.environ['CC'] = "clang" and os.environ['CXX'] = "clang++")

now my problem is when I try to import bolt I get this error:

>>> import bolt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/.local/lib/python2.7/site-packages/pybolt-0.1.4-py2.7-linux-x86_64.egg/bolt/__init__.py", line 5, in <module>
    from bolt_api import *  # noqa
  File "~/.local/lib/python2.7/site-packages/pybolt-0.1.4-py2.7-linux-x86_64.egg/bolt/bolt_api.py", line 6, in <module>
    import bolt  # inner bolt because of SWIG
  File "~/.local/lib/python2.7/site-packages/pybolt-0.1.4-py2.7-linux-x86_64.egg/bolt/bolt.py", line 17, in <module>
    _bolt = swig_import_helper()
  File "~/.local/lib/python2.7/site-packages/pybolt-0.1.4-py2.7-linux-x86_64.egg/bolt/bolt.py", line 16, in swig_import_helper
    return importlib.import_module('_bolt')
  File "~/anaconda2/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: ~/.local/lib/python2.7/site-packages/pybolt-0.1.4-py2.7-linux-x86_64.egg/_bolt.so: undefined symbol: _ZNKSt3__120__vector_base_commonILb1EE20__throw_length_errorEv

Armin Bashizade · Answer 2 · Wed Feb 14 2018 16:09:40 GMT+0800 (China Standard Time)

setting the environment variable LDFLAGS to -lc++ solved the problem. (adding os.environ['LDFLAGS'] = '-lc++' to setup.py)

dblalock · Answer 3 · Thu Feb 15 2018 10:49:17 GMT+0800 (China Standard Time)

I marked this as closed by adding your changes as per our email exchange. Thanks for your help in getting to the bottom of this. For those seeing this in the future, I'm not 100% certain this fix will work on all systems and so would be happy to reopen this if you have problems.

goobee · Answer 4 · Wed Sep 05 2018 18:11:38 GMT+0800 (China Standard Time)

I miss another problem look like this
python/bolt/../../cpp/src/include/public.hpp:23: Warning 315: Nothing known about 'std::vector'.
python/bolt/config.i:45: Warning 315: Nothing known about 'Eigen::Matrix'.
python/bolt/config.i:46: Warning 315: Nothing known about 'Eigen::Dynamic'.
python/bolt/config.i:46: Warning 315: Nothing known about 'Eigen::Dynamic'.
python/bolt/config.i:47: Warning 315: Nothing known about 'Eigen::RowMajor'.
python/bolt/config.i:45: Warning 315: Nothing known about 'Eigen::Matrix'.
python/bolt/config.i:46: Warning 315: Nothing known about 'Eigen::Dynamic'.
python/bolt/config.i:46: Warning 315: Nothing known about 'Eigen::Dynamic'.
python/bolt/config.i:47: Warning 315: Nothing known about 'Eigen::RowMajor'.
clang -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DNDEBUG=1 -I/home/public/anaconda3/lib/python3.6/site-packages/numpy/core/include -Icpp/src/quantize -Icpp/src/utils -Icpp/src/include -Icpp/src/external -Icpp/src/external/eigen -I/home/yhx/bolt/python/bolt -I/home/public/anaconda3/include/python3.6m -c python/bolt/native_wrap.cpp -o build/temp.linux-x86_64-3.6/python/bolt/native_wrap.o -std=c++14 -fno-rtti -march=native -ffast-math -std=c++11 -stdlib=libc++
python/bolt/native_wrap.cpp:3053:10: fatal error: 'stdexcept' file not found
#include
^
1 error generated.
error: command 'clang' failed with exit status 1

dblalock · Answer 5 · Fri Sep 07 2018 10:15:49 GMT+0800 (China Standard Time)

This might be an issue with the stdlib that you're using. Can you build the following simple program?

main.cpp:

#include <stdexcept>
int main() { return 0; }

with the command:

$ clang++ main.cpp -o main.out -std=c++14 -stdlib=libc++

I also just removed the duplicate std= flags from the configuration, which may have caused problems.

goobee · Answer 6 · Fri Sep 07 2018 10:22:40 GMT+0800 (China Standard Time)

i got this by using clang++ ,and g++ is ok

root@sci:/home/xx# clang++ main.cpp -o main.out -std=c++14 -stdlib=libc++ main.cpp:1:9: fatal error: 'stdexcept' file not found #include<stdexcept> ^ 1 error generated. root@sci:/home/xx# root@sci:/home/xx# root@sci:/home/xx# g++ main.cpp

dblalock · Answer 7 · Fri Sep 07 2018 12:45:29 GMT+0800 (China Standard Time)

Great. Now what happens when you instead use a different stdlib? I.e.:

$ clang++ main.cpp -o main.out -std=c++14 -stdlib=libstdc++

or

$ clang++ main.cpp -o main.out -std=c++14

Also, what do you get when you do:

$ clang -v

goobee · Answer 8 · Sun Sep 09 2018 11:50:29 GMT+0800 (China Standard Time)

hello,when i follow those two command,

$ clang++ main.cpp -o main.out -std=c++14 -stdlib=libstdc++
$ clang++ main.cpp -o main.out -std=c++14

I got the same error

[root@localhost bolt]# clang++ main.cpp -o main.out -std=c++14     -stdlib=libstdc++
In file included from main.cpp:1:
In file included from /usr/bin/../lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/stdexcept:39:
In file included from /usr/bin/../lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/string:40:
In file included from /usr/bin/../lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/char_traits.h:40:
In file included from /usr/bin/../lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/postypes.h:40:
In file included from /usr/bin/../lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/cwchar:44:
/usr/include/wchar.h:39:11: fatal error: 'stdarg.h' file not found
# include <stdarg.h>
1 error generated.


[root@localhost bolt]# clang -v
clang version 3.9.1 (tags/RELEASE_391/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

dblalock · Answer 9 · Sun Sep 09 2018 13:49:11 GMT+0800 (China Standard Time)

Hmm. Based on what's here, can you compile a one-line cpp file that just contains:

 #include <cstdarg>

Also, if I'm counting the ..s properly, it appears to be searching in /usr/include/c++/4.8.5/ for all of your headers. This issue suggests that this header is supposed to get added when you install clang. Do you see stdarg.h in that directory? If not, what about when you do:

$ find /usr/include -name 'stdarg.h'

Also, for the sake of others seeing this thread, I edited your post so the error messages appear as code and the "include" doesn't look like a header. Hope that's okay. (The backticks didn't work across the line boundaries.)

Scott H. Hawley · Answer 10 · Wed Sep 01 2021 12:48:54 GMT+0800 (China Standard Time)

Tried to get this to work on Colab, but got "unterminated string constant" build error from swig after cd bolt && python setup.py install:

swigging cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i to cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake_wrap.cpp
swig -python -c++ -o cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake_wrap.cpp cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i
cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i:3: Error: Unterminated string constant
error: command 'swig' failed with exit status 1

My minimal Colab example link is here: https://colab.research.google.com/drive/1ATN8-R4xqghk9faNBqxFoE28iKd7WCZp?usp=sharing

zz ji · Answer 11 · Fri Sep 03 2021 16:34:23 GMT+0800 (China Standard Time)

Tried to get this to work on Colab, but got "unterminated string constant" build error from swig after cd bolt && python setup.py install:
swigging cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i to cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake_wrap.cpp
swig -python -c++ -o cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake_wrap.cpp cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i
cpp/src/external/eigen/.hg/store/data/_c_test_config.cmake.i:3: Error: Unterminated string constant
error: command 'swig' failed with exit status 1
My minimal Colab example link is here: https://colab.research.google.com/drive/1ATN8-R4xqghk9faNBqxFoE28iKd7WCZp?usp=sharing

I encounter this error with my Ubuntu likewise.
It seem to be caused by the upgrade of swig's version up to 4.0 (my swig version in ubuntu is 4.0).But I am not sure.
THere is a similar bug:
InsightSoftwareConsortium/ITK#2026

husisy · Answer 12 · Sat Sep 04 2021 23:19:30 GMT+0800 (China Standard Time)

I guess the folder cpp/src/external/eigen/.hg is managed by the software Mercurial (something like git) which will lead to the issue Unterminated string constant, so I simply delete this folder to go on the installation.

Then, I struggle with the issue "command clang not exist" on Ubuntu. To solve this, I turn to run this code on Mac.

Also, in setup.py, some items included in the list srcFiles will cause swig to fail, so I add modified several lines in it like below

exclude_file_ext = ['.cxxlist','.cu','.cmake','.in','.css','.entry','.evaluator','.expression','.main','.preamble','.traits']
srcFiles += [x for x in files if not any(x.endswith(y) for y in exclude_file_ext)]

After all above, it seems start to build the project, but fail agian...... The error is like below and I have no idea now...

  In file included from cpp/src/external/eigen-trimmed/unsupported/doc/examples/PolynomialSolver1.cpp:1:
  In file included from cpp/src/external/eigen-trimmed/unsupported/Eigen/Polynomials:40:
  cpp/src/external/eigen-trimmed/unsupported/Eigen/src/Polynomials/PolynomialSolver.h:365:43: error: no member named 'abs' in namespace 'Eigen::numext'; did you mean simply 'abs'?
            if( internal::isMuchSmallerThan(numext::abs(numext::imag(m_roots[i])),
                                            ^~~~~~~~~~~
                                            abs
  cpp/src/external/eigen-trimmed/Eigen/src/Core/GlobalFunctions.h:58:36: note: 'abs' declared here
    EIGEN_ARRAY_DECLARE_GLOBAL_UNARY(abs,scalar_abs_op)
                                     ^
  In file included from cpp/src/external/eigen-trimmed/unsupported/doc/examples/PolynomialSolver1.cpp:1:
  In file included from cpp/src/external/eigen-trimmed/unsupported/Eigen/Polynomials:40:
  cpp/src/external/eigen-trimmed/unsupported/Eigen/src/Polynomials/PolynomialSolver.h:366:43: error: no member named 'abs' in namespace 'Eigen::numext'; did you mean simply 'abs'?
                                            numext::abs(numext::real(m_roots[i])),
                                            ^~~~~~~~~~~
                                            abs
  cpp/src/external/eigen-trimmed/Eigen/src/Core/GlobalFunctions.h:58:36: note: 'abs' declared here
    EIGEN_ARRAY_DECLARE_GLOBAL_UNARY(abs,scalar_abs_op)
                                     ^
  In file included from cpp/src/external/eigen-trimmed/unsupported/doc/examples/PolynomialSolver1.cpp:1:
  In file included from cpp/src/external/eigen-trimmed/unsupported/Eigen/Polynomials:40:

Scott H. Hawley · Answer 13 · Sun Sep 05 2021 03:11:51 GMT+0800 (China Standard Time)

On Ubuntu, after following @husisy's advice and removing eigen's .hg/ folder, I start getting "unknown file type" errors, such as:

error: unknown file type '.cmake' (from 'cpp/src/external/eigen/CTestConfig.cmake')

^on Colab, and on my Pop_OS (Ubuntu variant) laptop I get:

error: unknown file type '.cu' (from 'cpp/src/external/eigen-trimmed/unsupported/test/cxx11_tensor_argmax_gpu.cu')

EDIT: After modifying setup.py as per @husisy, I'm able to get it down to an error about a missing file that apparently only exists on Apple systems:

clang -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fdebug-prefix-map=/build/python3.7-LSlbJj/python3.7-3.7.11=. -fstack-protector-strong -Wformat -Werror=format-security -g -fdebug-prefix-map=/build/python3.7-LSlbJj/python3.7-3.7.11=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DNDEBUG=1 -I/usr/local/lib/python3.7/dist-packages/numpy/core/include -Icpp/src/utils -Icpp/src/external -Icpp/src/include -Icpp/src/quantize -Icpp/src/external/eigen -Icpp/src/external/eigen-trimmed -Icpp/src/external/eigen/blas -Icpp/src/external/eigen/unsupported -Icpp/src/external/eigen/debug -Icpp/src/external/eigen/demos -Icpp/src/external/eigen/cmake -Icpp/src/external/eigen/doc -Icpp/src/external/eigen/scripts -Icpp/src/external/eigen/bench -Icpp/src/external/eigen/Eigen -Icpp/src/external/eigen/failtest -Icpp/src/external/eigen/blas/fortran -Icpp/src/external/eigen/unsupported/Eigen -Icpp/src/external/eigen/unsupported/Eigen/src -Icpp/src/external/eigen/unsupported/Eigen/CXX11 -Icpp/src/external/eigen/unsupported/Eigen/src/SparseExtra -Icpp/src/external/eigen/unsupported/Eigen/src/Splines -Icpp/src/external/eigen/unsupported/Eigen/src/NumericalDiff -Icpp/src/external/eigen/unsupported/Eigen/src/AutoDiff -Icpp/src/external/eigen/unsupported/Eigen/src/NonLinearOptimization -Icpp/src/external/eigen/unsupported/Eigen/src/Skyline -Icpp/src/external/eigen/unsupported/Eigen/src/IterativeSolvers -Icpp/src/external/eigen/unsupported/Eigen/src/KroneckerProduct -Icpp/src/external/eigen/unsupported/Eigen/src/LevenbergMarquardt -Icpp/src/external/eigen/unsupported/Eigen/src/SpecialFunctions -Icpp/src/external/eigen/unsupported/Eigen/src/MatrixFunctions -Icpp/src/external/eigen/unsupported/Eigen/src/FFT -Icpp/src/external/eigen/unsupported/Eigen/src/Polynomials -Icpp/src/external/eigen/unsupported/Eigen/src/BVH -Icpp/src/external/eigen/unsupported/Eigen/src/MoreVectorization -Icpp/src/external/eigen/unsupported/Eigen/src/Eigenvalues -Icpp/src/external/eigen/unsupported/Eigen/src/EulerAngles -Icpp/src/external/eigen/unsupported/Eigen/src/SpecialFunctions/arch -Icpp/src/external/eigen/unsupported/Eigen/src/SpecialFunctions/arch/CUDA -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src/Tensor -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src/ThreadPool -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src/TensorSymmetry -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src/util -Icpp/src/external/eigen/unsupported/Eigen/CXX11/src/TensorSymmetry/util -Icpp/src/external/eigen/debug/msvc -Icpp/src/external/eigen/debug/gdb -Icpp/src/external/eigen/demos/mix_eigen_and_c -Icpp/src/external/eigen/demos/opengl -Icpp/src/external/eigen/demos/mandelbrot -Icpp/src/external/eigen/doc/snippets -Icpp/src/external/eigen/doc/special_examples -Icpp/src/external/eigen/doc/examples -Icpp/src/external/eigen/bench/spbench -Icpp/src/external/eigen/bench/btl -Icpp/src/external/eigen/bench/tensors -Icpp/src/external/eigen/bench/perf_monitoring -Icpp/src/external/eigen/bench/btl/generic_bench -Icpp/src/external/eigen/bench/btl/cmake -Icpp/src/external/eigen/bench/btl/libs -Icpp/src/external/eigen/bench/btl/data -Icpp/src/external/eigen/bench/btl/actions -Icpp/src/external/eigen/bench/btl/generic_bench/timers -Icpp/src/external/eigen/bench/btl/generic_bench/utils -Icpp/src/external/eigen/bench/btl/generic_bench/init -Icpp/src/external/eigen/bench/btl/generic_bench/static -Icpp/src/external/eigen/bench/btl/libs/ublas -Icpp/src/external/eigen/bench/btl/libs/blaze -Icpp/src/external/eigen/bench/btl/libs/mtl4 -Icpp/src/external/eigen/bench/btl/libs/tensors -Icpp/src/external/eigen/bench/btl/libs/BLAS -Icpp/src/external/eigen/bench/btl/libs/blitz -Icpp/src/external/eigen/bench/btl/libs/eigen2 -Icpp/src/external/eigen/bench/btl/libs/eigen3 -Icpp/src/external/eigen/bench/btl/libs/STL -Icpp/src/external/eigen/bench/btl/libs/tvmet -Icpp/src/external/eigen/bench/btl/libs/gmm -Icpp/src/external/eigen/bench/perf_monitoring/gemm -Icpp/src/external/eigen/Eigen/src -Icpp/src/external/eigen/Eigen/src/QR -Icpp/src/external/eigen/Eigen/src/SVD -Icpp/src/external/eigen/Eigen/src/CholmodSupport -Icpp/src/external/eigen/Eigen/src/OrderingMethods -Icpp/src/external/eigen/Eigen/src/PaStiXSupport -Icpp/src/external/eigen/Eigen/src/LU -Icpp/src/external/eigen/Eigen/src/IterativeLinearSolvers -Icpp/src/external/eigen/Eigen/src/plugins -Icpp/src/external/eigen/Eigen/src/Jacobi -Icpp/src/external/eigen/Eigen/src/Core -Icpp/src/external/eigen/Eigen/src/Householder -Icpp/src/external/eigen/Eigen/src/PardisoSupport -Icpp/src/external/eigen/Eigen/src/SparseQR -Icpp/src/external/eigen/Eigen/src/misc -Icpp/src/external/eigen/Eigen/src/MetisSupport -Icpp/src/external/eigen/Eigen/src/UmfPackSupport -Icpp/src/external/eigen/Eigen/src/SuperLUSupport -Icpp/src/external/eigen/Eigen/src/Cholesky -Icpp/src/external/eigen/Eigen/src/SparseCholesky -Icpp/src/external/eigen/Eigen/src/StlSupport -Icpp/src/external/eigen/Eigen/src/SPQRSupport -Icpp/src/external/eigen/Eigen/src/Eigenvalues -Icpp/src/external/eigen/Eigen/src/Geometry -Icpp/src/external/eigen/Eigen/src/SparseCore -Icpp/src/external/eigen/Eigen/src/SparseLU -Icpp/src/external/eigen/Eigen/src/LU/arch -Icpp/src/external/eigen/Eigen/src/Core/arch -Icpp/src/external/eigen/Eigen/src/Core/products -Icpp/src/external/eigen/Eigen/src/Core/functors -Icpp/src/external/eigen/Eigen/src/Core/util -Icpp/src/external/eigen/Eigen/src/Core/arch/SSE -Icpp/src/external/eigen/Eigen/src/Core/arch/AltiVec -Icpp/src/external/eigen/Eigen/src/Core/arch/CUDA -Icpp/src/external/eigen/Eigen/src/Core/arch/AVX512 -Icpp/src/external/eigen/Eigen/src/Core/arch/AVX -Icpp/src/external/eigen/Eigen/src/Core/arch/Default -Icpp/src/external/eigen/Eigen/src/Core/arch/NEON -Icpp/src/external/eigen/Eigen/src/Core/arch/ZVector -Icpp/src/external/eigen/Eigen/src/Geometry/arch -Icpp/src/external/eigen-trimmed/unsupported -Icpp/src/external/eigen-trimmed/Eigen -Icpp/src/external/eigen-trimmed/unsupported/test -Icpp/src/external/eigen-trimmed/unsupported/doc -Icpp/src/external/eigen-trimmed/unsupported/bench -Icpp/src/external/eigen-trimmed/unsupported/Eigen -Icpp/src/external/eigen-trimmed/unsupported/test/mpreal -Icpp/src/external/eigen-trimmed/unsupported/doc/snippets -Icpp/src/external/eigen-trimmed/unsupported/doc/examples -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11 -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/SparseExtra -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/Splines -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/NumericalDiff -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/AutoDiff -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/NonLinearOptimization -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/Skyline -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/IterativeSolvers -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/KroneckerProduct -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/LevenbergMarquardt -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/SpecialFunctions -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/MatrixFunctions -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/FFT -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/Polynomials -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/BVH -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/MoreVectorization -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/Eigenvalues -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/EulerAngles -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/SpecialFunctions/arch -Icpp/src/external/eigen-trimmed/unsupported/Eigen/src/SpecialFunctions/arch/GPU -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src/Tensor -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src/ThreadPool -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src/TensorSymmetry -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src/util -Icpp/src/external/eigen-trimmed/unsupported/Eigen/CXX11/src/TensorSymmetry/util -Icpp/src/external/eigen-trimmed/Eigen/src -Icpp/src/external/eigen-trimmed/Eigen/src/QR -Icpp/src/external/eigen-trimmed/Eigen/src/SVD -Icpp/src/external/eigen-trimmed/Eigen/src/CholmodSupport -Icpp/src/external/eigen-trimmed/Eigen/src/OrderingMethods -Icpp/src/external/eigen-trimmed/Eigen/src/PaStiXSupport -Icpp/src/external/eigen-trimmed/Eigen/src/LU -Icpp/src/external/eigen-trimmed/Eigen/src/IterativeLinearSolvers -Icpp/src/external/eigen-trimmed/Eigen/src/plugins -Icpp/src/external/eigen-trimmed/Eigen/src/Jacobi -Icpp/src/external/eigen-trimmed/Eigen/src/Core -Icpp/src/external/eigen-trimmed/Eigen/src/Householder -Icpp/src/external/eigen-trimmed/Eigen/src/PardisoSupport -Icpp/src/external/eigen-trimmed/Eigen/src/SparseQR -Icpp/src/external/eigen-trimmed/Eigen/src/misc -Icpp/src/external/eigen-trimmed/Eigen/src/MetisSupport -Icpp/src/external/eigen-trimmed/Eigen/src/UmfPackSupport -Icpp/src/external/eigen-trimmed/Eigen/src/SuperLUSupport -Icpp/src/external/eigen-trimmed/Eigen/src/Cholesky -Icpp/src/external/eigen-trimmed/Eigen/src/SparseCholesky -Icpp/src/external/eigen-trimmed/Eigen/src/StlSupport -Icpp/src/external/eigen-trimmed/Eigen/src/SPQRSupport -Icpp/src/external/eigen-trimmed/Eigen/src/Eigenvalues -Icpp/src/external/eigen-trimmed/Eigen/src/Geometry -Icpp/src/external/eigen-trimmed/Eigen/src/SparseCore -Icpp/src/external/eigen-trimmed/Eigen/src/SparseLU -Icpp/src/external/eigen-trimmed/Eigen/src/LU/arch -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch -Icpp/src/external/eigen-trimmed/Eigen/src/Core/products -Icpp/src/external/eigen-trimmed/Eigen/src/Core/functors -Icpp/src/external/eigen-trimmed/Eigen/src/Core/util -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/SSE -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/AltiVec -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/CUDA -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/AVX -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/Default -Icpp/src/external/eigen-trimmed/Eigen/src/Core/arch/NEON -Icpp/src/external/eigen-trimmed/Eigen/src/Geometry/arch -I/content/bolt/python/bolt -I/usr/include/python3.7m -c python/bolt/native_wrap.cpp -o build/temp.linux-x86_64-3.7/python/bolt/native_wrap.o -std=c++14 -fno-rtti -march=native -ffast-math -stdlib=libc++
python/bolt/native_wrap.cpp:3053:10: fatal error: 'stdexcept' file not found
#include <stdexcept>
         ^~~~~~~~~~~
1 error generated.
error: command 'clang' failed with exit status 1

Colab link is the same as before: https://colab.research.google.com/drive/1ATN8-R4xqghk9faNBqxFoE28iKd7WCZp?usp=sharing#scrollTo=ks4orqb0Sxrs

Scott H. Hawley · Answer 14 · Wed Sep 08 2021 00:52:25 GMT+0800 (China Standard Time)

Update: Above Colab goes significantly further than before but still fails.
'Fixed' the above Apple-specific error after reading further up this Issue (woops) by creating a blank file. Then spent about 7 hours on Saturday. Seems that clang is getting called where clang++ should be, and that bolt code has hard-coded "eigen/Core" where it should read (and other libraries use) "Eigen/Core" or "eigen3/Eigen/Core". Is this an artifact using a case-unspecific filesystem like Mac?

In file included from python/bolt/native_wrap.cpp:3129:
In file included from /content/bolt/python/bolt/../../cpp/src/include/public.hpp:19:
In file included from cpp/src/external/eigen/Eigen/Core:96:
In file included from /usr/include/c++/7.5.0/complex:45:
In file included from /usr/include/c++/7.5.0/sstream:38:
In file included from /usr/include/c++/7.5.0/istream:38:
In file included from /usr/include/c++/7.5.0/ios:42:
In file included from /usr/include/c++/7.5.0/bits/ios_base.h:46:
/usr/include/c++/7.5.0/system_error:94:13: error: unknown type name '__cow_string'
    virtual __cow_string
            ^
/usr/include/c++/7.5.0/system_error:341:36: error: expected class name
  class system_error : public std::runtime_error
                                   ^
/usr/include/c++/7.5.0/system_error:348:7: error: member initializer 'runtime_error' does not name a non-static data member or base class
    : runtime_error(__ec.message()), _M_code(__ec) { }
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/7.5.0/system_error:351:7: error: member initializer 'runtime_error' does not name a non-static data member or base class
    : runtime_error(__what + ": " + __ec.message()), _M_code(__ec) { }
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/7.5.0/system_error:354:7: error: member initializer 'runtime_error' does not name a non-static data member or base class
    : runtime_error(__what + (": " + __ec.message())), _M_code(__ec) { }
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/7.5.0/system_error:360:7: error: member initializer 'runtime_error' does not name a non-static data member or base class
    : runtime_error(error_code(__v, __ecat).message()),
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/7.5.0/system_error:364:7: error: member initializer 'runtime_error' does not name a non-static data member or base class
    : runtime_error(__what + ": " + error_code(__v, __ecat).message()),
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
python/bolt/native_wrap.cpp:3141:12: fatal error: 'eigen/Core' file not found
  #include "eigen/Core"
           ^~~~~~~~~~~~
8 errors generated.

@dblalock Let us know when we can try this out! Looking forward to it. Until then, 👋

Chengtao An · Answer 15 · Wed Sep 08 2021 10:44:07 GMT+0800 (China Standard Time)

Hello, when I install the bolt on my macbook, I follow the installing process but meet the following problem.
swigging cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in.i to cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in_wrap.cpp swig -python -c++ -o cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in_wrap.cpp cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in.i cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in.i:1: Error: Illegal token '?'. cpp/src/external/eigen/.hg/store/data/_c_test_custom.cmake.in.i:1: Error: Syntax error in input(1). error: command 'swig' failed with exit status 1
Even changing the version of swig can not solve this problem. So I ask what can I do? Thanks.

dblalock · Answer 16 · Wed Sep 08 2021 12:22:25 GMT+0800 (China Standard Time)

Thanks for everyone's feedback and work on this (special thanks to @drscotthawley, as well as @husisy and @calvinmccarter, who've put a bunch of time into helping to get all this working). I'm looking into this, but it's kind of slow going at the moment (especially since I'm doing about 70h/week at my day job and don't have much time left over). I appreciate everyone's enthusiasm and patience.

Calvin McCarter · Answer 17 · Wed Sep 08 2021 20:20:53 GMT+0800 (China Standard Time)

@dblalock With a bit more documentation around the C++ API, I'd be happy to write a basic command-line interface. Obviously getting and returning matrices via file i/o is slow, but it would be sufficient for me.

mneilly · Answer 18 · Tue Oct 12 2021 09:42:33 GMT+0800 (China Standard Time)

FWIW - I have a branch that has changes so that things can be built and run on Linux and Mac. The changes basically add a CMakeLists.txt to build with cmake, add a kmc2 submodule that works with python3 and setup.py changes so the python package builds. There is a build.sh that was tested to work on Debian 11 and Mac OS X 10.14.6.

https://github.com/mneilly/bolt/tree/mneilly/fixes

Cristian Tatu · Answer 19 · Wed Oct 13 2021 22:47:04 GMT+0800 (China Standard Time)

Now it works for me! Good job thank you
The only error I get now is when I use accuracy=high. With accuracy low and medium everything works fine.
this was my test

/home/ctatu/anaconda3/lib/python3.8/site-packages/pybolt-0.1.4-py3.8-linux-x86_64.egg/bolt/bolt_api.py:104: RuntimeWarning: invalid value encountered in true_divide
  seeds = kmc2.kmc2(X, k).astype(np.float32)
Traceback (most recent call last):
  File "tst.py", line 14, in <module>
    enc.fit(X)
  File "/home/ctatu/anaconda3/lib/python3.8/site-packages/pybolt-0.1.4-py3.8-linux-x86_64.egg/bolt/bolt_api.py", line 464, in fit
    centroids = _learn_centroids(X, ncentroids=ncentroids,
  File "/home/ctatu/anaconda3/lib/python3.8/site-packages/pybolt-0.1.4-py3.8-linux-x86_64.egg/bolt/bolt_api.py", line 140, in _learn_centroids
    centroids, labels = kmeans(X_in, ncentroids)
  File "/home/ctatu/anaconda3/lib/python3.8/site-packages/pybolt-0.1.4-py3.8-linux-x86_64.egg/bolt/bolt_api.py", line 104, in kmeans
    seeds = kmc2.kmc2(X, k).astype(np.float32)
  File "kmc2.pyx", line 97, in kmc2.kmc2 (kmc2.c:2646)
  File "mtrand.pyx", line 928, in numpy.random.mtrand.RandomState.choice
ValueError: probabilities contain NaN

dblalock · Answer 20 · Sun Jan 02 2022 10:19:42 GMT+0800 (China Standard Time)

@mneilly this is awesome, thanks! I'll be happy to merge it if you open a PR.

charlswzc · Answer 21 · Mon Apr 25 2022 15:29:11 GMT+0800 (China Standard Time)

@dblalock @mneilly If anybody runs BOLT perfectly, can you you guys share the information of your machine? Like OS version, swig version, and so on. Many thanks!

I tried win10, ubuntu 18.04/20.01, still not working. [sad]

mneilly · Answer 22 · Fri Apr 29 2022 23:59:30 GMT+0800 (China Standard Time)

@dblalock I opened the PR.
@charlswzc The PR has a Dockerfile which works that you can use as a reference.

windspirit95 · Answer 23 · Thu Jun 23 2022 13:08:26 GMT+0800 (China Standard Time)

@dblalock @mneilly If anybody runs BOLT perfectly, can you you guys share the information of your machine? Like OS version, swig version, and so on. Many thanks!

I tried win10, ubuntu 18.04/20.01, still not working. [sad]

Finally, I could install Bolt perfectly, with kmc2 install

Ubuntu version: 18.04
swig 3.0.12
Install Clang - Ubuntu

sudo apt install clang

Install Bolt:

git clone -b mneilly/fixes https://github.com/mneilly/bolt.git
cd bolt
python setup.py install
cd ../

Install kmc2

git clone -b mneilly/cythonize https://github.com/mneilly/kmc2.git
cd kmc2
python setup.py install
cd ../

Thanks to @mneilly for the PR ^^

dblalock · Answer 24 · Sat Jun 25 2022 07:21:04 GMT+0800 (China Standard Time)

Yes, big thanks to mneilly! For any future readers, I actually merged this branch into Bolt, so in theory you should get the same results in the main repo now.

That said, I would never bet on the robustness of mostly-unmaintained code spanning two languages with legacy FFI interfaces and x86-specific libraries, so worth giving mneilly's branch a try if this repo doesn't work. The docker container he wrote (see BUILD.md should be more reliable.