Installation Errors with macOS Big Sur

Question

Installation Errors with macOS Big Sur

bernardopacini opened this issue 4 years ago · comments

Hi,

I recently updated my machine to Mac OS Big Sur and find that I am no longer able to install F90wrap. At first I found an issue with Numpy that threw an error and sorted that out using the recent Numpy commit (found here: https://github.com/numpy/numpy/commit/1ccb4c6dbfa6194d1627885e39f81823bce44fc7).

Unfortunately, even with this change I am not able to install F90wrap. It appears to be an issue with not finding a library needed by gfortran. I wonder if you have insight on why this might be happening and if you have had luck with F90wrap on Big Sur.

This is the message I run into:

    compiling Fortran sources
    Fortran f77 compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
    Fortran f90 compiler: /usr/local/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
    Fortran fix compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
    creating build/temp.macosx-11-x86_64-3.9/f90wrap
    compile options: '-Ibuild/src.macosx-11-x86_64-3.9/build/src.macosx-11-x86_64-3.9/f90wrap -I/Users/bernardo_pacini/Documents/T-PINN/env/lib/python3.9/site-packages/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/bernardo_pacini/Documents/T-PINN/env/include -I/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c'
    gfortran:f90: f90wrap/sizeoffortran.f90
    gfortran: warning: couldn't understand version 11
    /usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle build/temp.macosx-11-x86_64-3.9/build/src.macosx-11-x86_64-3.9/f90wrap/sizeof_fortran_tmodule.o build/temp.macosx-11-x86_64-3.9/build/src.macosx-11-x86_64-3.9/build/src.macosx-11-x86_64-3.9/f90wrap/fortranobject.o build/temp.macosx-11-x86_64-3.9/f90wrap/sizeoffortran.o -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -lgfortran -o build/lib.macosx-11-x86_64-3.9/f90wrap/sizeof_fortran_t.cpython-39-darwin.so
    gfortran: warning: couldn't understand version 11
    ld: library not found for -lbundle1.o
    collect2: error: ld returned 1 exit status
    error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle build/temp.macosx-11-x86_64-3.9/build/src.macosx-11-x86_64-3.9/f90wrap/sizeof_fortran_tmodule.o build/temp.macosx-11-x86_64-3.9/build/src.macosx-11-x86_64-3.9/build/src.macosx-11-x86_64-3.9/f90wrap/fortranobject.o build/temp.macosx-11-x86_64-3.9/f90wrap/sizeoffortran.o -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0/../../.. -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -lgfortran -o build/lib.macosx-11-x86_64-3.9/f90wrap/sizeof_fortran_t.cpython-39-darwin.so" failed with exit status 1
    ----------------------------------------

I can provide my complete installation log if it would be helpful.

Thank you very much in advance,
-Bernardo

James Kermode · Answer 1 · Fri Dec 18 2020 15:40:24 GMT+0800 (China Standard Time)

Unfortunately this is a a gfortran problem not an f90wrap one. I’ve also recently updated to Big Sur and haven’t managed to restore a fully functional environment yet. If you are using homebrew, start by updating all packages, but I think we may need to wait for a gcc/gfortran patch.

Bernardo Pacini · Answer 2 · Fri Dec 18 2020 22:43:48 GMT+0800 (China Standard Time)

Thank you for the quick response! I have the packages updated so I'll keep waiting for the gcc/gfortran patch. Thanks again.

James Kermode · Answer 3 · Wed Jan 06 2021 23:59:38 GMT+0800 (China Standard Time)

I have this working now, here's what I set in my environment - may need adapting

export CPATH=`xcrun --show-sdk-path`/usr/include
export LDFLAGS=-L`xcrun --show-sdk-path`/usr/lib
export NPY_DISTUTILS_APPEND_FLAGS=1
export CC=gcc-10
export CXX-g++-10
export F90=gfortran-10
export F77=gfortran-10

Bernardo Pacini · Answer 4 · Thu Jan 07 2021 01:37:56 GMT+0800 (China Standard Time)

Thank you for the update. I tried that and unfortunately it still does not work for me. I am using GNU Fortran 10.2.0 and numpy 1.20.0rc2. Which version of numpy do you use?

This is the bottom snippet of the error I run into:

    File "/usr/local/lib/python3.9/site-packages/numpy/distutils/fcompiler/gnu.py", line 73, in gnu_version_match
      raise ValueError(err + version_string)
  ValueError: A valid Fortran version was not found in this string:
  gfortran-10: warning: couldn't understand version 11
  10.2.0
  
  ########### EXT COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : gcc
  
  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none
  
  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt._cache_write[796] : write cache to path -> /private/var/folders/82/x2lgxrz96clgm5p30pp_jxl40000gn/T/pip-install-qzlpt6t8/f90wrap_f86aba0605a545d4b27ef9e42c455b3d/build/temp.macosx-11-x86_64-3.9/ccompiler_opt_cache_ext.py
  ----------------------------------------
  ERROR: Failed building wheel for f90wrap

James Kermode · Answer 5 · Thu Jan 07 2021 01:42:30 GMT+0800 (China Standard Time)

This is what I have - a bit older numpy, not sure why

[jameskermode@moriarty f90wrap]$ python
Python 3.7.9 (default, Nov 20 2020, 23:58:42)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.17.1'

Bernardo Pacini · Answer 6 · Thu Jan 07 2021 01:50:43 GMT+0800 (China Standard Time)

Thank you. Using an older version of numpy (even the current stable one) I run into a type error for the OS version number. This is documented as an issue on numpy: numpy/numpy#18093. I am not sure why I see different results on my machine from what you have working now. I was under the impression that we would have to wait until gfortran-10 is able to find a version for OS 11.

Bernardo Pacini · Answer 7 · Fri Jan 29 2021 11:16:59 GMT+0800 (China Standard Time)

As an update to this issue:

With the newest Xcode command line tools and GCC 10.2.0_3 I am able to install f90wrap using numpy 1.20.0rc2. Unfortunately, I am not able to completely compile my code as I run into the following error

./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:898:1: error: use of undeclared identifier 'setjmpvalue'
setjmpvalue = setjmp(environment_buffer);
^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:899:5: error: use of undeclared identifier 'setjmpvalue'
if (setjmpvalue != 0) {
    ^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:1141:1: error: use of undeclared identifier 'setjmpvalue'
setjmpvalue = setjmp(environment_buffer);
^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:1142:5: error: use of undeclared identifier 'setjmpvalue'
if (setjmpvalue != 0) {
    ^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:1446:1: error: use of undeclared identifier 'setjmpvalue'
setjmpvalue = setjmp(environment_buffer);
^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:1447:5: error: use of undeclared identifier 'setjmpvalue'
if (setjmpvalue != 0) {
    ^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:1993:1: error: use of undeclared identifier 'setjmpvalue'
setjmpvalue = setjmp(environment_buffer);
^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:1994:5: error: use of undeclared identifier 'setjmpvalue'
if (setjmpvalue != 0) {
    ^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2223:1: error: use of undeclared identifier 'setjmpvalue'
setjmpvalue = setjmp(environment_buffer);
^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2224:5: error: use of undeclared identifier 'setjmpvalue'
if (setjmpvalue != 0) {
    ^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2336:1: error: use of undeclared identifier 'setjmpvalue'
setjmpvalue = setjmp(environment_buffer);
^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2337:5: error: use of undeclared identifier 'setjmpvalue'
if (setjmpvalue != 0) {
    ^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2440:1: error: use of undeclared identifier 'setjmpvalue'
setjmpvalue = setjmp(environment_buffer);
^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2441:5: error: use of undeclared identifier 'setjmpvalue'
if (setjmpvalue != 0) {
    ^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2547:1: error: use of undeclared identifier 'setjmpvalue'
setjmpvalue = setjmp(environment_buffer);
^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2548:5: error: use of undeclared identifier 'setjmpvalue'
if (setjmpvalue != 0) {
    ^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2655:1: error: use of undeclared identifier 'setjmpvalue'
setjmpvalue = setjmp(environment_buffer);
^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2656:5: error: use of undeclared identifier 'setjmpvalue'
if (setjmpvalue != 0) {
    ^
./src.macosx-11-x86_64-3.9/_helix_pyf90module.c:2759:1: error: use of undeclared identifier 'setjmpvalue'
setjmpvalue = setjmp(environment_buffer);
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
1 warning and 20 errors generated.
1 warning generated.
error: Command "clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -DNPY_DISABLE_OPTIMIZATION=1 -I./src.macosx-11-x86_64-3.9/./src.macosx-11-x86_64-3.9 -I/usr/local/lib/python3.9/site-packages/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/tcl-tk/include -I/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c ./src.macosx-11-x86_64-3.9/_helix_pyf90module.c -o ./src.macosx-11-x86_64-3.9/_helix_pyf90module.o -MMD -MF ./src.macosx-11-x86_64-3.9/_helix_pyf90module.o.d" failed with exit status 1

I wonder if you know of a solution for this.

Thank you again in advance!

Bernardo Pacini · Answer 8 · Fri Jan 29 2021 13:07:00 GMT+0800 (China Standard Time)

I am able to circumvent this issue by switching to the normal f2py command instead of f2py-f90wrap, however I then end up with a flat namespace that cannot be imported into Python.

ImportError: dlopen(/Users/bernardo_pacini/Documents/HELIX/python_src/_helix_pyf90.so, 2): Symbol not found: _f90wrap_abort_
  Referenced from: /Users/bernardo_pacini/Documents/HELIX/python_src/_helix_pyf90.so
  Expected in: flat namespace
 in /Users/bernardo_pacini/Documents/HELIX/python_src/_helix_pyf90.so

Thanks again in advance for any feedback.

James Kermode · Answer 9 · Fri Jan 29 2021 16:58:51 GMT+0800 (China Standard Time)

OK, leave this with me, that should be fixable as its surely just an undeclared variable that gcc is now being more strict about.

James Kermode · Answer 10 · Fri Jan 29 2021 16:59:38 GMT+0800 (China Standard Time)

For the latter error, you need to remove calls to the abort function if you are using standard f2py rather than f2py-f90wrap.

Bernardo Pacini · Answer 11 · Sat Jan 30 2021 10:02:20 GMT+0800 (China Standard Time)

Great, thank you!

With regards to the abort calls, is that something I am supposed to manually remove from the generated code or is there an option / flag I should set? I do not call abort explicitly from any of my code so I believe those references are automatically generated. Regardless, I will stick with f2py-f90wrap in the future. Thanks again.

Bernardo Pacini · Answer 12 · Sun Jan 31 2021 08:20:01 GMT+0800 (China Standard Time)

I was able to compile my code by changing the f2py-f90wrap script on line 114. As it stands now, it is:

numpy.f2py.rules.routine_rules['body'] = numpy.f2py.rules.routine_rules['body'].replace('\tvolatile int f2py_success = 1;\n', """\tvolatile int f2py_success = 1;
\tint setjmpvalue; /* James Kermode - for setjmp */
""")

Changing it to:

numpy.f2py.rules.routine_rules['body'] = numpy.f2py.rules.routine_rules['body'].replace("    volatile int f2py_success = 1;\n", """    volatile int f2py_success = 1;
    int setjmpvalue; /* James Kermode - for setjmp */
""")

works properly. On my machine it seems like the first version does not recognize the string to replace, so int jmpvalue is never added in the C file. This may be due to the \t character being interpreted differently now on Mac, but I am not sure of what exactly could have changed.

I am not submitting this as a PR as I am not sure it is a standard fix for all machines, it simply worked on my machine.

I also am finding strange behavior once compiled in two ways:
1 - lines that are longer than the limit specified for --py-max-line-length PY_MAX_LINE_LENGTH are broken up in a way that is not correct in python and then the import fails due to indentation mistakes. I fixed this by increasing the PY_MAX_LINE_LENGTH value, but of course that is a unique fix for this case. This may fall into the same issue as above, as tabs may be somehow handled differently now.
2- the precision of the Fortran code is not maintained, with functions such as sqrt() truncating precision. I have not looked into this carefully, but this is not an issue I have run into in the past. I think this is unlikely to be related to f90wrap and instead is a Fortran issue with the recent updates.

I have not run into these issues when using Ubuntu 20.04, GCC 9, and Python 3.8, so they seem to be unique to Mac and newer versions of Python + GCC.

James Kermode · Answer 13 · Tue Feb 02 2021 00:37:34 GMT+0800 (China Standard Time)

Thanks for the update. Would removing the whitespace completely from both arguments to the replace() call be a more robust solution? If that works locally for you perhaps you could submit it as a PR?

The other two issues that you report are indeed strange; let me know if you discover the underlying cause(s).

James Kermode · Answer 14 · Tue Feb 02 2021 00:39:28 GMT+0800 (China Standard Time)

With regards to the abort calls, is that something I am supposed to manually remove from the generated code or is there an option / flag I should set? I do not call abort explicitly from any of my code so I believe those references are automatically generated. Regardless, I will stick with f2py-f90wrap in the future. Thanks again.

If you remove the -a/--abort-func from the f90wrap command line then I think these calls should not longer be inserted; if they are this is a separate bug so please open a new issue.

Bernardo Pacini · Answer 15 · Tue Feb 02 2021 10:47:04 GMT+0800 (China Standard Time)

I submitted a PR with the update to make the rule replacement statement more general.

I checked the -a/--abort-func tags and I do not have those in my function call but I still run into errors. I will open a new issue.

James Kermode · Answer 16 · Tue Feb 02 2021 17:58:37 GMT+0800 (China Standard Time)

Can this issue be closed now?

Bernardo Pacini · Answer 17 · Wed Feb 03 2021 01:13:03 GMT+0800 (China Standard Time)

I tested this on Ubuntu and the fix works there as well. Closing.

Ananth Sridharan · Answer 18 · Sun Feb 14 2021 06:56:53 GMT+0800 (China Standard Time)

@bernardopacini @jameskermode :
I tried out this fix, works on my Mac as well, so long as all fortran memory is present within modules.

One issue that I noticed (maybe unrelated?) is that I'm no longer able to instantiate derived types from Python anymore; system throws an error:
zsh: illegal hardware instruction python3 driver.py

Bernardo Pacini · Answer 19 · Sun Feb 14 2021 11:42:11 GMT+0800 (China Standard Time)

I have not run into that issue. Do you have a code snippet as an example? I initialize derived types from python, but my use-case may be different from yours.

Ananth Sridharan · Answer 20 · Sun Feb 14 2021 12:49:44 GMT+0800 (China Standard Time)

I tried a reduced example - it was related to mac OS-specific stack size limits (was able to instantiate successfully when trying smaller sizes of arrays of types). My test code is attached - its got some handy CMake wrappers to call f90wrap (works for Mac, Ubuntu (various versions) and WSL Ubuntu).

Ananth Sridharan · Answer 21 · Mon Feb 15 2021 05:01:36 GMT+0800 (China Standard Time)

github.zip
To trigger the error, set
nfc = 20
(instead of 10) in common/src/bemt/Source/pyint_modules/precision.F90

Compile as follows (assuming you start at the main folder after unzipping)

cd cmake; mkdir build; cd build
FC=gfortran cmake ../../ && make -j

Run with
python3 driver.py in main folder

James Kermode · Answer 22 · Mon Feb 15 2021 05:07:54 GMT+0800 (China Standard Time)

Please can you open a new issue for this?