`zig c++` does not include `stdc++`, `clang++` does

Question

`zig c++` does not include `stdc++`, `clang++` does

folkertdev opened this issue 3 years ago · comments

I hit a case where zig c++ is not a drop-in replacement for a clang that uses the equivalent llvm version (llvm 12, zig 0.8.0, clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-20.04)

This command works

clang++ -v main.cc.o  -o onnx2c   libonnx2c_lib.a /usr/local/lib/libprotobuf.so

while this does not

zig c++ -v main.cc.o  -o onnx2c   libonnx2c_lib.a /usr/local/lib/libprotobuf.so

The zig verson reports missing c++ stdlib symbols, e.g.

ld.lld: error: undefined symbol: std::ostream::put(char)
>>> referenced by graph.cc
>>>               graph.cc.o:(toC::LSTM::print_activation(std::ostream&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const) in archive libonnx2c_lib.a
>>> referenced by graph.cc
>>>               graph.cc.o:(toC::LSTM::print_lstm_kernel(std::ostream&, bool) const) in archive libonnx2c_lib.a
>>> referenced by graph.cc
>>>               graph.cc.o:(toC::LSTM::print_lstm_kernel(std::ostream&, bool) const) in archive libonnx2c_lib.a
>>> referenced 89 more times

Here are the full linker invocations:

Clang:

"/usr/bin/ld" --hash-style=both --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o onnx2c /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/10/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/10 -L/usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/x86_64-linux-gnu/../../lib64 -L/usr/lib/gcc/x86_64-linux-gnu/10/../../.. -L/home/folkertdev/Downloads/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-20.04/bin/../lib -L/lib -L/usr/lib main.cc.o libonnx2c_lib.a /usr/local/lib/libprotobuf.so -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/10/crtend.o /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/crtn.o

Zig c++

ld.lld -error-limit=0 -z stack-size=16777216 --gc-sections --eh-frame-hdr -m elf_x86_64 -o /home/folkertdev/.cache/zig/o/10863395297406fc6a07a3d66e5dae94/onnx2c /home/folkertdev/.cache/zig/o/d72a08251764ca3b50ad6f9c2178502c/Scrt1.o /home/folkertdev/.cache/zig/o/fd2f7ed75d6a8bb131dbeb450b22976c/crti.o -L /usr/local/lib64 -L /usr/local/lib -L /usr/lib/x86_64-linux-gnu -L /lib64 -L /lib -L /usr/lib64 -L /usr/lib -L /lib/x86_64-linux-gnu -dynamic-linker /lib64/ld-linux-x86-64.so.2 main.cc.o libonnx2c_lib.a /usr/local/lib/libprotobuf.so /home/folkertdev/.cache/zig/o/8d7f892320208e3eb85beca5cbadba3b/libcompiler_rt.a /home/folkertdev/.cache/zig/o/c1bb35f3336ba0415196640af079cbb6/libc++abi.a /home/folkertdev/.cache/zig/o/498c83bc6b6945fe94a3be48eb4ce9e0/libc++.a /home/folkertdev/.cache/zig/o/668363e59260ab7aeb85271c2fd416e5/libunwind.a /home/folkertdev/.cache/zig/o/68130f6cc2832b2537f2ccc0002f97c2/libm.so.6 /home/folkertdev/.cache/zig/o/68130f6cc2832b2537f2ccc0002f97c2/libpthread.so.0 /home/folkertdev/.cache/zig/o/68130f6cc2832b2537f2ccc0002f97c2/libc.so.6 /home/folkertdev/.cache/zig/o/68130f6cc2832b2537f2ccc0002f97c2/libdl.so.2 /home/folkertdev/.cache/zig/o/68130f6cc2832b2537f2ccc0002f97c2/librt.so.1 /home/folkertdev/.cache/zig/o/68130f6cc2832b2537f2ccc0002f97c2/libld.so.2 /home/folkertdev/.cache/zig/o/68130f6cc2832b2537f2ccc0002f97c2/libutil.so.1 /home/folkertdev/.cache/zig/o/b61066544a773e7c5cad2d3678dfea62/libc_nonshared.a /home/folkertdev/.cache/zig/o/3b50ad42196c5bd14aad1360daafeb7d/crtn.o

The difference that makes a difference is these arguments that clang does give to the linker, and zig does not:

    -L/usr/lib/gcc/x86_64-linux-gnu/10\
    -lstdc++\

The zig linker invocation does include

/home/folkertdev/.cache/zig/o/c1bb35f3336ba0415196640af079cbb6/libc++abi.a\
/home/folkertdev/.cache/zig/o/498c83bc6b6945fe94a3be48eb4ce9e0/libc++.a\

But that is apparently not enough? Also I tried

zig c++ -L/usr/lib/gcc/x86_64-linux-gnu/10 -v main.cc.o -o onnx2c libonnx2c_lib.a /usr/local/lib/libprotobuf.so -lstdc++

(include the gnu path and the stdc++ flag) in the zig c++ command, but the -lstdc++ flag is not present in the linker invocation. So

is this a bug, or am I missing some c++ detail here?
is there a way to force -lstdc++ to make it to the linker, either with zig c++ or zig build-exe

Michael Dusan · Answer 1 · Sat Sep 25 2021 00:14:20 GMT+0800 (China Standard Time)

Which object contains graph.cc.o ? I'm going to guess: libprotobuf.so and it has a dependency on libstdc++.so. To verify this dep, ldd /usr/local/lib/libprotobuf.so.

No single executable/binary should use 2 different c++ libraries.

The solve to this is to build protobuf (and any of its dependencies that are c++ releated) with zig c++.

Folkert de Vries · Answer 2 · Sat Sep 25 2021 01:30:53 GMT+0800 (China Standard Time)

no, graph.cc is contained in libonnx2c_lib.a. That .a is constructed completely by zig. But if I understand you correctly building protobuf with zig c++ would still solve the issue?

Michael Dusan · Answer 3 · Sat Sep 25 2021 01:33:13 GMT+0800 (China Standard Time)

protobuf might be another issue here but let's push that aside for now. How is libonnx2c_lib.a built? Can you show the command-line for building some .o for it?

Folkert de Vries · Answer 4 · Sat Sep 25 2021 01:55:25 GMT+0800 (China Standard Time)

Sure, here is what currently works

zig build-lib src/graph.cc src/graph_print.cc src/node.cc src/onnx.pb.cc src/tensor.cc src/util.cc -lc++  -I/home/folkertdev/tg/scailable/sclbl-onnx-to-c/build -isystem /home/folkertdev/tg/scailable/sclbl-onnx-to-c/./src -I/home/folkertdev/tg/scailable/sclbl-onnx-to-c/src -I/home/folkertdev/tg/scailable/sclbl-onnx-to-c/src/utils -I/usr/local/include -I/home/folkertdev/tg/scailable/sclbl-onnx-to-c/aixlog/include -cflags -std=c++11 -O3 -Wall -Wl, -Wno-misleading-indentation -Wno-error=unused-variable     -- -femit-bin=all.cc.o

zig ar qc libonnx2c_lib.a all.cc.o 

zig ranlib libonnx2c_lib.a

clang++ -I/home/folkertdev/tg/scailable/sclbl-onnx-to-c/build -isystem /home/folkertdev/tg/scailable/sclbl-onnx-to-c/./src  -O3 -Wall -Wno-misleading-indentation -Wno-error=unused-variable    -I/home/folkertdev/tg/scailable/sclbl-onnx-to-c/src -I/home/folkertdev/tg/scailable/sclbl-onnx-to-c/src/utils -I/home/folkertdev/tg/scailable/sclbl-onnx-to-c/aixlog/include -Wall -Wno-unused-variable -std=c++11 -o main.cc.o -c src/main.cc

clang++ -v main.cc.o  -o onnx2c   libonnx2c_lib.a /usr/local/lib/libprotobuf.so

Now with your information, I tried switching both of those clang++ to zig c++ (had only tried them independently so far). That seems to make some progress, but ultimately hits a bunch of errors like this one:

ld.lld: error: duplicate symbol: toC::Node::onnx_ir_version
>>> defined at node.cc
>>>            node.cc.o:(toC::Node::onnx_ir_version) in archive libonnx2c_lib.a
>>> defined at node.cc
>>>            node.cc.o:(.bss+0x0) in archive libonnx2c_lib.a

Michael Dusan · Answer 5 · Sat Sep 25 2021 02:17:35 GMT+0800 (China Standard Time)

re: zig build-lib ... can you check if any of the -I or -isystem args supply a directory which may supply c++ include files. Looking to rule out conflict. _ZIG_/lib/libcxx/include is where the proper files are found.

Also this is a bit tricky, would like to see the include dir search list used for .cpp → .o

remove local zig-cache
reduce the zig build-lib line to only use src/graph.cc and remove the remaining .cc files from command and add --verbose-cc
cut-n-paste the huge command line for .cpp → .o, which should be something like zig clang ... and append to that command-line -v

so something like this:

rm -fr zig-cache
zig build-lib src/graph.cc -lc++ ..... --verbose-cc
zig clang ..... -v

which should yield something like this:

clang -cc1 version 12.0.1 based upon LLVM 12.0.1 default target x86_64-unknown-linux-gnu
#include "..." search starts here:
#include <...> search starts here:
 /home/mike/project/zig/work/main/lib/libcxx/include
 /home/mike/project/zig/work/main/lib/libcxxabi/include
 /home/mike/project/zig/work/main/lib/libunwind/include
 /home/mike/project/zig/work/main/lib/include
 /home/mike/project/zig/work/main/lib/libc/include/x86_64-linux-gnu
 /home/mike/project/zig/work/main/lib/libc/include/generic-glibc
 /home/mike/project/zig/work/main/lib/libc/include/x86_64-linux-any
 /home/mike/project/zig/work/main/lib/libc/include/any-linux-any
End of search list.

Andrew Kelley · Answer 6 · Sat Sep 25 2021 02:19:31 GMT+0800 (China Standard Time)

C++ pretends to have an ABI, but it's a lie. That's the crucial problem here.

Zig provides libc++, however the libc++ that zig provides cannot be mixed with any C++ objects compiled from other compilers. If you do, you get undefined behavior. So you can't use zig c++ on libonnx2c_lib.a or /usr/local/lib/libprotobuf.so because these C++ objects were compiled with something other than zig c++.

I'm upset that C++ works this way but there is nothing we can do about it.

zig c++ is a drop-in C++ compiler if you use it for all your C++ objects; if not, it does not help you.

Folkert de Vries · Answer 7 · Sat Sep 25 2021 02:30:00 GMT+0800 (China Standard Time)

To clarify, even combining output of zig build-obj and zig c++ would run into trouble?

Andrew Kelley · Answer 8 · Sat Sep 25 2021 02:31:23 GMT+0800 (China Standard Time)

If the input file were C++ code, yes. You can learn about more intricacies of C++ to find out what versions of compilers do or don't work together, or what features of the language are affected by this ABI mismatch. But the simplest way to remember it is that you have to use the same C++ compiler for all C++ code.

Andrew Kelley · Answer 9 · Sat Sep 25 2021 02:32:22 GMT+0800 (China Standard Time)

The upshot here is that if you have libonnx2c_lib.a or /usr/local/lib/libprotobuf.so on your system then these were presumably built by the system C++ compiler, so you should be able to rely on that to build your other C++ objects.

Folkert de Vries · Answer 10 · Sat Sep 25 2021 02:37:57 GMT+0800 (China Standard Time)

yes, the system c++ compiler works fine. I just would have liked to terminate a CMakeLists.txt and use build.zig instead, and that now seems like it won't work out

Andrew Kelley · Answer 11 · Sat Sep 25 2021 04:44:39 GMT+0800 (China Standard Time)

If we want to support that use case, it will require zig build to additionally support being an external compiler driver. That's something that may very well be in scope of the project, but it is not currently implemented.