Build failing to redirect to C++11

Question

Build failing to redirect to C++11

MichaelChirico opened this issue 4 years ago · comments

I'm still quite poor at configure scripts so I'll do what I can to help. Here's the install.packages output:

* installing *source* package ‘RcppSimdJson’ ...
** package ‘RcppSimdJson’ successfully unpacked and MD5 sums checked
** using staged installation
** setting up C++17
** libs
Error in .shlib_internal(args) : 
  C++17 standard requested but CXX17 is not defined

the ** setting up C++17 I think is coming from configure, but IINM it shouldn't be reaching that branch. Here's my R CMD config --all | grep CXX:

CXX = g++ -std=gnu++11
CXXFLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXXPICFLAGS = -fpic
CXX11 = g++
CXX11STD = -std=gnu++11
CXX11FLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXX11PICFLAGS = -fpic
CXX14 = g++
CXX14STD = -std=gnu++14
CXX14FLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXX14PICFLAGS = -fpic
CXX17 =
CXX17STD =
CXX17FLAGS =
CXX17PICFLAGS =
CXXCPP = g++ -std=gnu++11 -E
CXX98 = g++
CXX98STD = -std=gnu++98
CXX98FLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXX98PICFLAGS = -fpic

Based on this, this line in configure looks funny to me:

CXX17=`${R_HOME}/bin/R CMD config --all | awk '{print $1}' | grep "^CXX17$"`

What's the purpose of this line? Seems to me it will always return CXX17 whether or not there's anything after =...

Something along this lines would seem to be better?

awk '/^CXX17 =/ {print $0}'

Michael Chirico · Answer 1 · Mon Jul 13 2020 17:19:28 GMT+0800 (China Standard Time)

Another note -- I was able to compile a simple program against -std=g++17, even though R CMD config seems to be unaware. Does this mean that R is simply unaware of the capabilities of g++? Is there any way to update that? Or is there something deeper going on:

echo 'int main() {}' > 'test.c'
g++ -std=gnu++17 test.c -o test.o

Dirk Eddelbuettel · Answer 2 · Mon Jul 13 2020 20:15:02 GMT+0800 (China Standard Time)

Two or three things in the avalanche of information:

Your awk one liner seems better. I do those at time too, not sure what I was thinking there. But see all the comments above in configure and use e.g. git blame to assert this was a first pass at configure. Apparently I had errors under what was then r-devel. Which some people may use as their R for years to come.
I have configure snippets elsewhere, some going back almost twenty years (!!) in the case of RQuantLib's, trying to get the g++ version from its version string to assert a minimum version. That is imperfect. I think I'll add something to configure to actually compile a mininmal example and see what C++ version it returns.
Of course g++ and R can diverge! R simply remembers what it was compiled with. On systems where you get binaries that may be a very different version than you have now, that you upgraded to since, or other reasons. It's a simple heuristic for a minimal version we can assume. See 2. above :)

So yes, this needs work, so thanks for the reminder! Especially now that we have parsing code enticing folks like you to try it in production we need to shore this up. Will work on it.

Michael Chirico · Answer 3 · Mon Jul 13 2020 20:20:19 GMT+0800 (China Standard Time)

OK great. I'm really quite new at this so glad I'm not totally off my rocker. I take 9t the point of R CMD javareconf is to "re-link" R with what's available on the system. is there anything like that for R / anyway manually to set what shows up in R CMD config (i.e. post build)?

This is beyond the scope of this issue, maybe I should ask elsewhere...

Dirk Eddelbuettel · Answer 4 · Mon Jul 13 2020 20:23:46 GMT+0800 (China Standard Time)

is there anything like that for R / anyway manually to set what shows up in R CMD config (i.e. post build)

Not to my 20+ years of reading Writing R Extensions and Inst and Admin. Note that the command you are thinking of is called R CMD javareconf and not C++ was hurt or engaged in its use... Java, as we all know, "is special". And the config file is immutable. You yourself set CXX (and friends) to direct it; it provides a fallback if you don't.

Michael Chirico · Answer 5 · Tue Jul 14 2020 02:07:54 GMT+0800 (China Standard Time)

OK some progress but still no luck building. I don't really know what the output's telling me this time.

First I added this to ~/.R/Makevars:

CXX17 = g++
CXX17STD = -std=gnu++17
CXX17FLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g

Then install.packages('RcppSimdJson') gives the following:

* installing *source* package ‘RcppSimdJson’ ...
** package ‘RcppSimdJson’ successfully unpacked and MD5 sums checked
** using staged installation
** setting up C++17
** libs
g++ -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG  -I"/databricks/spark/R/lib/Rcpp/include"  -DSIMDJSON_NO_COMPUTED_GOTO -I../inst/include -fopenmp   -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG  -I"/databricks/spark/R/lib/Rcpp/include"  -DSIMDJSON_NO_COMPUTED_GOTO -I../inst/include -fopenmp   -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c deserialize.cpp -o deserialize.o
In file included from ../inst/include/RcppSimdJson/deserialize/simplify.hpp:5:0,
                 from ../inst/include/RcppSimdJson/deserialize.hpp:5,
                 from ../inst/include/RcppSimdJson.hpp:5,
                 from deserialize.cpp:1:
../inst/include/RcppSimdJson/deserialize/../common.hpp:8:20: fatal error: optional: No such file or directory
compilation terminated.
/usr/lib/R/etc/Makeconf:177: recipe for target 'deserialize.o' failed

Dirk Eddelbuettel · Answer 6 · Tue Jul 14 2020 02:09:57 GMT+0800 (China Standard Time)

Can you tell us more about your platform? We seem to mostly work at CRAN apart from oldrel. If you are "blessed" with C@nt&s at work ... I feel for you.

Michael Chirico · Answer 7 · Tue Jul 14 2020 02:13:09 GMT+0800 (China Standard Time)

This is in a Databricks cluster. Not super familiar with all the details. I can ask about anything you have in mind. Sys.info() doesn't have anything exciting in it:

                                     sysname 
                                      "Linux" 
                                      release 
                          "4.15.0-1050-azure" 
                                      version 
"#55-Ubuntu SMP Sat Jun 29 00:27:54 UTC 2019"

What I think is relevant from sessionInfo():

R version 3.6.2 (2019-12-12)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.6 LTS

Matrix products: default
BLAS:   /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.18.so

It seems I spoke too soon about -std=gnu++17 working so easily:

https://stackoverflow.com/q/38253971/3576984

Would it make sense to put that part of the common.hpp header inside an IFDEF and redirect to <experimental/optional> as suggested? Or is this just likely to be the beginning of other problems later on...

Michael Chirico · Answer 8 · Tue Jul 14 2020 02:27:10 GMT+0800 (China Standard Time)

OK more progress. I followed this to install a proper compiler:

https://askubuntu.com/a/887791/362864

and replaced CXX17=g++ with CXX17=g++-7 in my ~/.R/Makevars.

Build output is:

* installing *source* package ‘RcppSimdJson’ ...
** package ‘RcppSimdJson’ successfully unpacked and MD5 sums checked
** using staged installation
** setting up C++17
** libs
g++-7 -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG  -I"/databricks/spark/R/lib/Rcpp/include"  -DSIMDJSON_NO_COMPUTED_GOTO -I../inst/include -fopenmp   -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c RcppExports.cpp -o RcppExports.o
g++-7 -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG  -I"/databricks/spark/R/lib/Rcpp/include"  -DSIMDJSON_NO_COMPUTED_GOTO -I../inst/include -fopenmp   -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c deserialize.cpp -o deserialize.o
g++-7 -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG  -I"/databricks/spark/R/lib/Rcpp/include"  -DSIMDJSON_NO_COMPUTED_GOTO -I../inst/include -fopenmp   -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c rcppsimdjson_utils_check.cpp -o rcppsimdjson_utils_check.o
g++-7 -std=gnu++17 -I"/usr/share/R/include" -DNDEBUG  -I"/databricks/spark/R/lib/Rcpp/include"  -DSIMDJSON_NO_COMPUTED_GOTO -I../inst/include -fopenmp   -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c simdjson_example.cpp -o simdjson_example.o
g++-7 -std=gnu++17 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o RcppSimdJson.so RcppExports.o deserialize.o rcppsimdjson_utils_check.o simdjson_example.o -fopenmp -L/usr/lib/R/lib -lR
/usr/bin/ld: RcppExports.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
RcppExports.o: error adding symbols: Bad value
collect2: error: ld returned 1 exit status

Dirk Eddelbuettel · Answer 9 · Tue Jul 14 2020 02:31:36 GMT+0800 (China Standard Time)

@knapply found gcc-7 to be dodgy too. Our first test matrix only covered g++ 8, 9, and 10, but we now include 7 and it passes there for us. You could try to loog at the Travis log and see what's different.

We should likely offer you a "forced" downgrade to C++11. So far we had relied on @lemire doing all the magic in the real code.

But given that you are "R package"-savvy, do the following:

fetch the .tar.gz
neuter/remove/rename the shell script configure
copy src/Makevars.in to src/Makevars and set C++11 in there explicitly.

That should do the trick.

Michael Chirico · Answer 10 · Tue Jul 14 2020 02:38:32 GMT+0800 (China Standard Time)

OK. Will try that tomorrow (somewhat easier said than done -- I don't have an interactive terminal 😓 ) to see if it at least works. Then I would try and figure out a way to automate that all in an init script...

knapply · Answer 11 · Tue Jul 14 2020 04:09:13 GMT+0800 (China Standard Time)

In #38 (comment)...

/usr/bin/ld: RcppExports.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC

Specifically, recompile with -fPIC.

My guess is that Rcpp was compiled with different flags than what's trying to compile RcppSimdJson.

Looking at #38 (comment) you have...

CXX = g++ -std=gnu++11
CXXFLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXXPICFLAGS = -fpic
CXX11 = g++
CXX11STD = -std=gnu++11
CXX11FLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXX11PICFLAGS = -fpic
CXX14 = g++
CXX14STD = -std=gnu++14
CXX14FLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXX14PICFLAGS = -fpic
CXX17 =
CXX17STD =
CXX17FLAGS =
CXX17PICFLAGS =
CXXCPP = g++ -std=gnu++11 -E
CXX98 = g++
CXX98STD = -std=gnu++98
CXX98FLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXX98PICFLAGS = -fpic

Specifically the CXX*PICFLAGS are...

CXXPICFLAGS = -fpic
CXX11PICFLAGS = -fpic
CXX98PICFLAGS = -fpic
CXX17PICFLAGS =

Can you try making CXX17PICFLAGS = -fpic like the others?

Dirk Eddelbuettel · Answer 12 · Tue Jul 14 2020 04:13:59 GMT+0800 (China Standard Time)

My guess is that Rcpp was compiled with different flags

My thinking too. Is that a databricks provided binary or yours? Anything weird happening with Makevars? I have never EVAR seen that link error with the CRAN Ubuntu binaries (i.e. the Michael Rutter builds of my Debian package) and PPAs.

Also, just to be sure, make sure R is set up as a dynamically linked build. I think CentOS played Crazy Uncle there for a while.

Worst case: get to work at home using Docker and sim the environment off a Ubuntu 18.04 (or 16.04 or what you have there) env. I have a few preloaded with various R packages in my various Docker Hub repos ...

Daniel Lemire · Answer 13 · Tue Jul 14 2020 04:16:42 GMT+0800 (China Standard Time)

Ping me if I can help. (I don't understand the issue, but if we can do something to help...)

knapply · Answer 14 · Tue Jul 14 2020 04:19:02 GMT+0800 (China Standard Time)

I found these w/ the same/similar error:

Dirk Eddelbuettel · Answer 15 · Tue Jul 14 2020 04:28:03 GMT+0800 (China Standard Time)

I do like myself an efficient search but allow me

case 1: unresolved, hence still "user error" aka pbkac
case 2: lacking -fPIC
case 3: see case 2
case 4: no idea, style file makes page unreadable
case 4: see case 2
case 5: user out of his breadth and does not know what he is doing (he had hit me elsewhere with his flashlight problem

So in short as we said before: ensure that R is built with dynamic library support, and maybe test with something simple. In case you had not built other things, the Rcpp::evalCpp("2 + 2") is still my favourite litmus test. If you don't see 4 your system is foobared and can't even build the simplest C++ routine adding two consts.

Then again methinks Michael would have hit roadblocks before this?

knapply · Answer 16 · Tue Jul 14 2020 04:46:10 GMT+0800 (China Standard Time)

I'd add Rcpp::evalCpp("2 + 2", plugins = "cpp17") for this instance. ;)

Dirk Eddelbuettel · Answer 17 · Tue Jul 14 2020 05:24:36 GMT+0800 (China Standard Time)

The likely culprit may be

CXX17PICFLAGS =

which is not what I have here where R came as the binary build by Michael based off the Debian package, and mirrored CRAN from his launchpad repo:

edd@rob:~$ grep PICFLAGS /etc/R/Makeconf 
CPICFLAGS = -fpic
CXXPICFLAGS = -fpic
CXX11PICFLAGS = -fpic
CXX14PICFLAGS = -fpic
CXX17PICFLAGS = -fpic
CXX20PICFLAGS = -fpic
DYLIB_LDFLAGS = -shared -fopenmp# $(CFLAGS) $(CPICFLAGS)
FPICFLAGS = -fpic
SHLIB_LDFLAGS = -shared# $(CFLAGS) $(CPICFLAGS)
ALL_CFLAGS =  $(PKG_CFLAGS) $(CPICFLAGS) $(SHLIB_CFLAGS) $(CFLAGS)
ALL_CXXFLAGS =  $(PKG_CXXFLAGS) $(CXXPICFLAGS) $(SHLIB_CXXFLAGS) $(CXXFLAGS)
ALL_OBJCFLAGS = $(PKG_OBJCFLAGS) $(CPICFLAGS) $(SHLIB_CFLAGS) $(OBJCFLAGS)
ALL_OBJCXXFLAGS = $(PKG_OBJCXXFLAGS) $(CXXPICFLAGS) $(SHLIB_CXXFLAGS) $(OBJCXXFLAGS)
ALL_FFLAGS = -fno-optimize-sibling-calls $(PKG_FFLAGS) $(FPICFLAGS) $(SHLIB_FFLAGS) $(FFLAGS)
ALL_FCFLAGS = -fno-optimize-sibling-calls $(P_FCFLAGS) $(FPICFLAGS) $(SHLIB_FFLAGS) $(FCFLAGS)
edd@rob:~$

Michael Chirico · Answer 18 · Tue Jul 14 2020 11:01:18 GMT+0800 (China Standard Time)

My guess is that Rcpp was compiled with different flags than what's trying to compile RcppSimdJson

Indeed true.

Rcpp was also built with CXX so I had to make some more edits to my ~/.R/Makevars as well, here's what I had that worked:

CXX = g++-7 -std=gnu++17
CXXFLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXXPICFLAGS = -fpic
CXX17 = g++-7
CXX17STD = -std=gnu++17
CXX17FLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXX17PICFLAGS = -fpic

(NB: I also added the CXX17PICFLAGS = -fpic, I'm not sure if that was also maybe crucial...)

With that and after remove.packages('Rcpp', .libPaths()), install.packages worked.

Michael Chirico · Answer 19 · Tue Jul 14 2020 11:02:31 GMT+0800 (China Standard Time)

A different question -- will I need to re-install all my other reverse depends of Rcpp as well? I've got 54 of those in my local library, not sure how many are required for the users on the cluster.

Dirk Eddelbuettel · Answer 20 · Tue Jul 14 2020 11:11:09 GMT+0800 (China Standard Time)

edits to my ~/.R/Makevars

So it was just your Makevars that was behind? Phew! That a) can happen and b) I was getting worried!

will I need to re-install all my other reverse depends

Not at all! The different C++ "standards" from within the same compiler are and remain interoperable. Just how gfortran and gcc and all the other frontends can mix as well! Only when the a) the compiler changes, or b) R internals changes do you need to. And that is ... rare, luckily.

Michael Chirico · Answer 21 · Tue Jul 14 2020 11:14:43 GMT+0800 (China Standard Time)

Hmm but notice that it's not the same compiler right?

Rcpp & its downstreams were built with g++, but now I've removed that and re-built with g++-7. Won't that break the other downstreams, just as RcppSimdJson was breaking because I was trying to build with g++-7 when Rcpp was built with g++?

Dirk Eddelbuettel · Answer 22 · Tue Jul 14 2020 11:16:50 GMT+0800 (China Standard Time)

They are interoperable too! Crazier still, you can also just switch to clang++ if you feel like and it still works. Magic, eh?

Michael Chirico · Answer 23 · Tue Jul 14 2020 11:26:26 GMT+0800 (China Standard Time)

Magic indeed. FWIW, I just restarted the cluster to test the necessity of the CXX flags.

Seems CXX17PICFLAGS alone was enough to break the logjam! Much easier and doesn't leave me feel uneasy about the reliability of magic 😎

Specifically these Makevars were sufficient:

CXX17 = g++-9
CXX17STD = -std=gnu++17
CXX17FLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXX17PICFLAGS = -fpic

Notice I also had a drop-in replacement for above to install g++-9 instead of g++-7.

Going to close since I think this thread is enough for any other passersby to hopefully figure out what to do. The summary in case they get lost in the noise:

My build was failing because I didn't have a C++17-compatible compiler installed. I was able to install one on my old Ubuntu 16.04 system with:

add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get update
apt-get install -y gcc-9 g++-9

Then I needed to tell R to use this compiler to build C++17 sources (such as RcppSimdJson). The way to do this is to edit the file at ~/.R/Makevars, you can append this:

CXX17 = g++-9
CXX17STD = -std=gnu++17
CXX17FLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
CXX17PICFLAGS = -fpic

After that, install.packages('RcppSimdJson') worked fine.