Using new compilers on linux fails with "Failed to import the site module"

Question

Using new compilers on linux fails with "Failed to import the site module"

Zaharid opened this issue 6 years ago · comments

I want to update this recipe:

https://github.com/Zaharid/conda-recipes/tree/master/lhapdf

to use the new compiler infrastructure. I rewrite the meta.yaml file this way:

package:
    name: lhapdf
    version: "6.2.1"

source:
    url: http://www.hepforge.org/archive/lhapdf/LHAPDF-6.2.1.tar.gz
    md5: 9e05567d538fdb4862d4781cd076d7db
    fn: LHAPDF-6.2.1.tar.gz

requirements:
    build:
        - {{compiler('cxx')}}
    host:
        - python
        - cython
    run:
        - python

build:
    number: 0
    detect_binary_files_with_prefix: True

test:
    commands:
        - lhapdf-config
        - python -c "import lhapdf"

about:
    home: https://lhapdf.hepforge.org/
    license: UNKNOWN

I get the following error when trying to build the recipe on a linux (Mint 18.3) laptop:

Failed to import the site module
Traceback (most recent call last):
  File "/home/zah/anaconda3/lib/python3.6/site.py", line 544, in <module>
    main()
  File "/home/zah/anaconda3/lib/python3.6/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/home/zah/anaconda3/lib/python3.6/site.py", line 282, in addusersitepackages
    user_site = getusersitepackages()
  File "/home/zah/anaconda3/lib/python3.6/site.py", line 258, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/home/zah/anaconda3/lib/python3.6/site.py", line 248, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/home/zah/anaconda3/lib/python3.6/sysconfig.py", line 601, in get_config_var
    return get_config_vars().get(name)
  File "/home/zah/anaconda3/lib/python3.6/sysconfig.py", line 550, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/home/zah/anaconda3/lib/python3.6/sysconfig.py", line 421, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata_x86_64_conda_cos6_linux_gnu'
Traceback (most recent call last):
  File "/home/zah/anaconda3/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 388, in main
    execute(sys.argv[1:])
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 379, in execute
    verify=args.verify)
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/api.py", line 187, in build
    need_source_download=need_source_download, config=config, variants=variants)
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/build.py", line 1801, in build_tree
    notest=notest,
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/build.py", line 1137, in build
    utils.check_call_env(cmd, env=env, cwd=src_dir)
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/utils.py", line 676, in check_call_env
    return _func_defaulting_env_to_os_environ(subprocess.check_call, *popenargs, **kwargs)
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/utils.py", line 672, in _func_defaulting_env_to_os_environ
    return func(_args, **kwargs)
  File "/home/zah/anaconda3/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/work/conda_build.sh']' returned non-zero exit status 1.

I am not sure where the error comes from. Seems to be some activate script. Note that it seems to be looking at the python executable in the root environment rather than the one that gets installed.

Full output:

$ conda build lhapdf/
Attempting to finalize metadata for lhapdf
INFO:conda_build.metadata:Attempting to finalize metadata for lhapdf
BUILD START: ['lhapdf-6.2.1-py36ha9410bc_0.tar.bz2']

The following NEW packages will be INSTALLED:

    binutils_impl_linux-64: 2.28.1-h04c84fa_2       
    binutils_linux-64:      7.2.0-hc67e822_14       
    ca-certificates:        2017.08.26-h1d4fec5_0   
    certifi:                2017.11.5-py36hf29ccca_0
    cython:                 0.26.1-py36h21c49d0_0   
    gcc_impl_linux-64:      7.2.0-hc5ce805_2        
    gcc_linux-64:           7.2.0-hf1c97a4_14       
    gxx_impl_linux-64:      7.2.0-hd3faf3d_2        
    gxx_linux-64:           7.2.0-h4bc98f9_14       
    libedit:                3.1-heed3624_0          
    libffi:                 3.2.1-hd88cf55_4        
    libgcc-ng:              7.2.0-h7cc24e2_2        
    libstdcxx-ng:           7.2.0-h7a57d05_2        
    ncurses:                6.0-h9df7e31_2          
    openssl:                1.0.2m-h26d622b_1       
    pip:                    9.0.1-py36h6c6f9ce_4    
    python:                 3.6.3-h6c0c0dc_5        
    readline:               7.0-ha6073c6_4          
    setuptools:             36.5.0-py36he42e2e1_0   
    sqlite:                 3.20.1-hb898158_2       
    tk:                     8.6.7-hc745277_3        
    wheel:                  0.30.0-py36hfd4bba0_1   
    xz:                     5.2.3-h55aa19d_2        
    zlib:                   1.2.11-ha838bed_2       

Source cache directory is: /home/zah/anaconda3/conda-bld/src_cache
Found source in cache: LHAPDF-6.2.1_9e05567d53.tar.gz
Extracting download
source tree in: /home/zah/anaconda3/conda-bld/lhapdf_1511638676021/work
INFO: activate-binutils_linux-64.sh made the following environmental changes:
+ADDR2LINE=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-addr2line
+AR=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-ar
+AS=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-as
+CXXFILT=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-c++filt
+ELFEDIT=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-elfedit
+GPROF=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-gprof
+HOST=x86_64-conda_cos6-linux-gnu
+LD_GOLD=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-ld.gold
+LD=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-ld
+NM=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-nm
+OBJCOPY=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-objcopy
+OBJDUMP=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-objdump
+RANLIB=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-ranlib
+READELF=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-readelf
+SIZE=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-size
+STRINGS=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-strings
+STRIP=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-strip
INFO: activate-gcc_linux-64.sh made the following environmental changes:
+CC=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-cc
+CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/include
+CPPFLAGS=-D_FORTIFY_SOURCE=2 -O2
+CPP=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-cpp
+DEBUG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -Og -g -Wall -Wextra -fcheck=all -fbacktrace -fimplicit-none -fvar-tracking-assignments -I/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/include
+GCC_AR=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-gcc-ar
+GCC=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-gcc
+GCC_NM=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-gcc-nm
+GCC_RANLIB=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-gcc-ranlib
+LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib -L/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib
+_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
INFO: activate-gxx_linux-64.sh made the following environmental changes:
+CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/include
+CXX=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-c++
+DEBUG_CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -Og -g -Wall -Wextra -fcheck=all -fbacktrace -fimplicit-none -fvar-tracking-assignments -I/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/include
+GXX=/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/bin/x86_64-conda_cos6-linux-gnu-g++
Failed to import the site module
Traceback (most recent call last):
  File "/home/zah/anaconda3/lib/python3.6/site.py", line 544, in <module>
    main()
  File "/home/zah/anaconda3/lib/python3.6/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/home/zah/anaconda3/lib/python3.6/site.py", line 282, in addusersitepackages
    user_site = getusersitepackages()
  File "/home/zah/anaconda3/lib/python3.6/site.py", line 258, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/home/zah/anaconda3/lib/python3.6/site.py", line 248, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/home/zah/anaconda3/lib/python3.6/sysconfig.py", line 601, in get_config_var
    return get_config_vars().get(name)
  File "/home/zah/anaconda3/lib/python3.6/sysconfig.py", line 550, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/home/zah/anaconda3/lib/python3.6/sysconfig.py", line 421, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata_x86_64_conda_cos6_linux_gnu'
Traceback (most recent call last):
  File "/home/zah/anaconda3/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 388, in main
    execute(sys.argv[1:])
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 379, in execute
    verify=args.verify)
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/api.py", line 187, in build
    need_source_download=need_source_download, config=config, variants=variants)
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/build.py", line 1801, in build_tree
    notest=notest,
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/build.py", line 1137, in build
    utils.check_call_env(cmd, env=env, cwd=src_dir)
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/utils.py", line 676, in check_call_env
    return _func_defaulting_env_to_os_environ(subprocess.check_call, *popenargs, **kwargs)
  File "/home/zah/anaconda3/lib/python3.6/site-packages/conda_build/utils.py", line 672, in _func_defaulting_env_to_os_environ
    return func(_args, **kwargs)
  File "/home/zah/anaconda3/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/home/zah/anaconda3/conda-bld/lhapdf_1511638676021/work/conda_build.sh']' returned non-zero exit status 1.

Nehal J Wani · Answer 1 · Sun Nov 26 2017 03:46:28 GMT+0800 (China Standard Time)

Could you try: conda update conda && conda update python in the environment from which you are trying to invoke conda build?

Zaharid · Answer 2 · Sun Nov 26 2017 03:51:47 GMT+0800 (China Standard Time)

Thanks! That seems to fix it. Does it mean that the defaults channel has to have higher priority than conda-forge for the compilers to work? Is it possible to use both channels?

Mike Sarahan · Answer 3 · Sun Nov 26 2017 04:07:39 GMT+0800 (China Standard Time)

The critical thing is that you must have defaults Python in your root environment. It might be best to keep conda forge out of your root environment, but add it into other envs. Each environment can have its own .condarc file. The need for this will go away over time, I think, as conda forge adopts conda build 3 and the new compilers.

…

On Nov 25, 2017 13:51, "Zaharid" ***@***.***> wrote: Thanks! That seems to fix it. Does it mean that the defaults channel has to have higher priority than conda-forge for the compilers to work? Is it possible to use both channels? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2523 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACV-c_HDb0njHrNj6WGbk7osB6YVuIYks5s6G_TgaJpZM4QqiN8> .

Mike Sarahan · Answer 4 · Sun Nov 26 2017 04:11:03 GMT+0800 (China Standard Time)

Sorry that's not quite accurate. Yes, right now, if conda forge is higher priority than defaults, building Python extensions with conda build will not work. There is extra stuff we include with Python to make it work, and if conda build create build envs with conda forge Python builds, that stuff will not be present.

…

On Nov 25, 2017 14:07, "Michael Sarahan" ***@***.***> wrote: The critical thing is that you must have defaults Python in your root environment. It might be best to keep conda forge out of your root environment, but add it into other envs. Each environment can have its own .condarc file. The need for this will go away over time, I think, as conda forge adopts conda build 3 and the new compilers. On Nov 25, 2017 13:51, "Zaharid" ***@***.***> wrote: > Thanks! That seems to fix it. Does it mean that the defaults channel has > to have higher priority than conda-forge for the compilers to work? Is > it possible to use both channels? > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#2523 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AACV-c_HDb0njHrNj6WGbk7osB6YVuIYks5s6G_TgaJpZM4QqiN8> > . >

Martin K. Scherer · Answer 5 · Tue Oct 09 2018 04:29:18 GMT+0800 (China Standard Time)

Somehow your suggested workaround does not work anymore @msarahan. For reference see https://travis-ci.org/markovmodel/PyEMMA/jobs/438821050 the root env is kept in the state from miniconda3 installer, and then the compiler is being dragged in by conda-build into the build environment. Is there any chance to make this work (again, as it used to work eg. last week or so).
"Mixing" Python from default and cf was not a problem on Linux. What has changed in the meantime?

Thanks a lot for looking into this.

Mike Sarahan · Answer 6 · Thu Oct 11 2018 22:20:59 GMT+0800 (China Standard Time)

I believe this is a conda-forge issue, as discussed at conda-forge/python-feedstock#203

I'm closing this issue, because I believe it is unrelated to conda-build. Please open a new issue if you would like to explain why you disagree.

Martin K. Scherer · Answer 7 · Thu Oct 11 2018 22:37:13 GMT+0800 (China Standard Time)

sorry for using this issue kind of off-topic.

github-actions · Answer 8 · Sun Mar 13 2022 14:07:14 GMT+0800 (China Standard Time)

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!