iwyu built against MacPorts LLVM-3.3 prelease can't find stdarg.h
vsapsai opened this issue · comments
Originally reported on Google Code with ID 100
What steps will reproduce the problem? Give the *exact* arguments passed
to include-what-you-use, and attach the input source file that gives the
problem (minimal test-cases are much appreciated!)
1. Run IWYU against the following file:
➜ iwyu clear
➜ iwyu cat ./cstdarg.cpp
#include <cstdarg>
int main(int argc, char* argv[]) {
return 0;
}.
What is the expected output? What do you see instead?
I'd expect this file to compile without warnings or errors. Potentially, IWYU might
tell me that cstdarg is not needed. However, what I actually get is a compilation error:
➜ iwyu include-what-you-use ./cstdarg.cpp
In file included from ./cstdarg.cpp:1:
/usr/include/c++/4.2.1/cstdarg:51:10: fatal error: 'stdarg.h' file not found
#include <stdarg.h>
^
/usr/include/c++/4.2.1/cstdarg should add these lines:
/usr/include/c++/4.2.1/cstdarg should remove these lines:
- #include <bits/c++config.h> // lines 50-50
The full include-list for /usr/include/c++/4.2.1/cstdarg:
---
(cstdarg.cpp has correct #includes/fwd-decls)
What version of the product are you using? On what operating system?
I'm using r472 on OS X 10.8 built against MacPorts LLVM 3.3 prerelease:
➜ iwyu port list clang-3.3 llvm-3.3
clang-3.3 @3.3-r180025 lang/llvm-3.3
llvm-3.3 @3.3-r180025 lang/llvm-3.3
Please provide any additional information below.
Reported by andrew.c.morrow on 2013-06-29 18:18:59
Can you compile cstdarg.cpp with clang-3.3? Looks like you are hitting http://clang.llvm.org/docs/FAQ.html#i-get-errors-about-some-headers-being-missing-stddef-h-stdarg-h
With current LLVM/Clang and IWYU I see the following output
/usr/include/c++/4.2.1/cstdarg should add these lines:
#include <cstddef> // for _GLIBCXX_BEGIN_NAMESPACE, etc
/usr/include/c++/4.2.1/cstdarg should remove these lines:
- #include <bits/c++config.h> // lines 50-50
The full include-list for /usr/include/c++/4.2.1/cstdarg:
#include <stdarg.h> // for va_end, va_list
#include <cstddef> // for _GLIBCXX_BEGIN_NAMESPACE, etc
---
(cstdarg.cpp has correct #includes/fwd-decls)
Reported by vsapsai on 2013-06-30 18:31:51
Volodymyr, I think IWYU is confused because your source file is named "cstdarg.cpp";
it considers "cstdarg" its associated header and tries to suggest improvements for
it.
I think the built-in include search is different on MSVC/Windows, so I can't reproduce
here.
Reported by kim.grasman on 2013-06-30 19:05:07
Since issue-99 had turned out to be resolved by rebuilding against the clang-3.3 stable,
I tried this one again. It still has problems. I also renamed the file from cstdarg.cpp
to blah.cpp to avoid the issue mentioned above. I still find that it errors out. However,
I found that if I explicitly include the include path to the llvm-3.3 directory containing
its stdarg.h, then all is well:
➜ cat ./blah.cpp
#include <cstdarg>
int main(int argc, char* argv[]) {
va_list list;
return 0;
}
➜ ./include-what-you-use ./blah.cpp
In file included from ./blah.cpp:1:
/usr/include/c++/4.2.1/cstdarg:51:10: fatal error: 'stdarg.h' file not found
#include <stdarg.h>
^
blah.cpp should add these lines:
blah.cpp should remove these lines:
- #include <cstdarg> // lines 1-1
The full include-list for blah.cpp:
---
Here it is with the additional include path:
➜ ./include-what-you-use ./blah.cpp -I/opt/local/libexec/llvm-3.3/lib/clang/3.3/include/
(blah.cpp has correct #includes/fwd-decls)
So this suggests to me that when building IWYU it isn't incorporating 'builtin' LLVM/clang
header paths somehow.
Reported by andrew.c.morrow on 2013-07-01 14:06:38
Per the link posted earlier, Clang searches for its builtin headers relative to the
binary. I think the same rule applies for IWYU.
This suggests you should move the include-what-you-use executable to the same directory
as clang, or copy the builtin includes and libs from /opt/local/libexec/llvm-33/lib/clang/3.3
to just next to include-what-you-use.
I still can't repro here, neither on Windows nor Ubuntu, but I'm not sure what headers
it finds and how. This is a bit of a problem with Clang, I think, that its header search
algorithm depends on what platform it was built on.
Note that if you build IWYU, Clang and LLVM together, this never becomes a problem,
because include-what-you-use will end up next to clang in the install tree.
I'd love to hear if you can get it to work by copying include and lib from clang/3.3,
and then we can think about a more transparent solution.
Reported by kim.grasman on 2013-07-01 19:02:36
This applies to LLVM 3.5 in MacPorts as well. Doing a regular build with clang++-mp-3.5
shows no problem, so it's iwyu related.
Reported by dsvensson on 2014-02-12 12:19:42
For what it's worth, I was running into the same problem with 3.4 and when I copied
include-what-you-use to the same directory as clang, the problem went away. So it appears
that the issue is caused by the headers that are shipped with clang not being resolved
correctly when include-what-you-use is located in a different directory.
Reported by davejohansen on 2014-04-24 20:27:54
I ran this by the Clang development list and it seems we need to distribute the builtin
headers with include-what-you-use.
Discussion here:
http://clang-developers.42468.n3.nabble.com/Standalone-tools-td4039043.html
And this is actually mentioned in passing here:
http://clang.llvm.org/docs/LibTooling.html#builtin-includes
I'll try and update our packaging script to include Clang's builtin headers.
Reported by kim.grasman on 2014-04-28 18:02:33
Do you want to add Clang's builtin headers to the distributed archive or to bake headers
into IWYU binary?
Reported by vsapsai on 2014-04-29 01:01:38
Since we have more than one build system, I thought I'd add the builtin headers to the
archive as a first step, but ideally I'd like to bake the headers in.
Reported by kim.grasman on 2014-04-29 04:03:30
OK, so I've experimented a little bit in my local Windows working copy.
It seems that building a directory tree like this works reasonably well:
include-what-you-use/
LICENSE.TXT
README
bin/
include-what-you-use.exe
fix_includes.py
lib/
clang/
3.5.0/
include/
<built-in headers>
I can copy this anywhere, add the include-what-you-use/bin root to my path in a clean
shell, and it works.
Does this look at all FHS-friendly?
@sylvestre on CC: How does this match what you're doing for Debian?
I wonder if we should add the equivalent of a make install step...?
Reported by kim.grasman on 2014-05-03 08:12:03
Actually, I am not sure how useful is a temporary fix by adding headers to distributed
archives. At least people in this thread haven't complained about distributed IWYU
binary, they've built IWYU manually.
I was thinking about creating a file containing baked in builtin headers with a Python
script. Reasons for that: I am not very good with build systems; there will be less
differences between different build systems.
Reported by vsapsai on 2014-05-05 21:35:03
They're building out-of-tree and so run into the same problem -- they can only get IWYU
to work if they put it in the Clang bin root.
But you're right; solving this in packaging won't help users building out-of-tree.
Yeah, a Python script sounds good, we should be able to run it from both CMake and
Make. The example from Olivier Goffart seemed pretty straightforward and should port
to Python easily.
I can build a draft implementation and we'll see how far it goes.
Reported by kim.grasman on 2014-05-06 09:01:07
I'm working on packaging IWYU for Fedora/RHEL so it can be easily installed using the
standard "yum install" process on those platforms. In that case, the header files are
already in the right place because both clang and IWYU are installed in the system
directories. So if an option is added to copy/package the headers as part of the install
process, then can an option to disable that please be added as well?
Thanks,
Dave
Reported by davejohansen on 2014-05-06 15:09:27
Kim, that looks great. Under Debian, I have clang installed here:
/usr/lib/llvm-3.5/lib/clang/3.5.0/include
Reported by sylvestre.ledru on 2014-05-06 15:40:22
I've worked on embedding the headers, but I can't get it to work properly.
After convincing CMake to run the Python script, I now get a compiler error from MSVC
that the string literals are larger than 64K. Since some of the builtin headers are
in the 3-500K range, this seems hard to get around.
I'll take a break from this and come back to it, but ideas are welcome!
Reported by kim.grasman on 2014-05-11 19:12:35
I just symlink the lib folder from my Clang install to a folder named lib where the
iwyu binary would expect it and it finds the builtins that way.
Reported by jason@citusdata.com on 2014-09-25 04:01:52
Is there any fix or work-around which does not require root? I don't have root access
to my machine and I would like to use IWYU.
I tried copying the folders from /usr/lib64/clang/3.5.0/* to my_iywu_folder/ but without
success.
Reported by ch.schmidbauer on 2014-12-04 13:04:05
The relative path distance between /usr/bin/clang and /usr/lib/clang/<ver>/ needs to
be preserved, so if you have the IWYU executable at ~/my_iwyu_folder/include-what-you-use
the builtins need to be at ~/my_iwyu_folder/../lib/clang/<ver>/.
Also, <ver> needs to match the version IWYU was built for, double-check with `include-what-you-use
--version` which Clang version it's built against.
Reported by kim.grasman on 2014-12-04 19:13:00
I built iwyu 0.3 with clang 3.5 on Fedora 21 and this issue seems to be fixed.
Reported by davejohansen on 2015-02-04 23:59:37
We haven't done anything actively, so chances are you've installed iwyu into /usr/bin
or built in-tree with Clang or had the builtin headers available in the right place
some other way.
Reported by kim.grasman on 2015-02-07 08:12:28
I'm not sure but I believe that this is a change in llvm/clang, because
with IWYU 0.2/clang 3.4 I had to set up the relative paths to get the tests
to pass, but with 0.3/3.5 the tests pass without the relative paths.
Reported by davejohansen on 2015-02-20 04:37:20
There has been zero progress on this issue for years. Is there a roadmap?
Sorry for unassigning, I have no idea why GH did this.
No definitive roadmap in sight.
The problem is that the headers belong to Clang and if it's not installed, the headers are not available.
Furthermore, if Clang is installed, IWYU needs to either know where or be installed in the same place, to be able to find them.
So in a sense, this is a packaging problem -- an IWYU package should depend on the Clang package, to ensure the headers are available. It also needs to install to the same bin path.
I've experimented with embedding the headers into the binary (see above), but haven't found anything that's portable.
A packager could maybe ship the headers with IWYU as well, but then we risk conflicting with Clang's headers, and that's probably not kosher/solvable on all platforms.
When building locally, you could build the clang-headers target, which I think copies them into the right location relative your build root.
Are these issues really all the same?
Which component hardcodes the search path?
In my case I'm building locally. Thus, the path information is available during the build. Why can't this not be hardcoded?
I don't know much about clang tooling, but even if the default search path can't be modified, an extra include would fix the issue just as well. Something like an -isystem option.
Good question. I'm not sure if they're all the same.
The search path for builtins appears to be called the "resource dir", and its default is hard-coded by the Clang Frontend library to point to some approximation of dirname(argv[0])/../lib/clang/<version> (see https://github.com/llvm-mirror/clang/blob/master/lib/Frontend/CompilerInvocation.cpp#L1693)
But there also seems to be a switch, -resource-dir, to override this default. If we could ship the builtin headers with IWYU and install them to a known location, we could make -resource-dir point to that location (something like $iwyu_bin/../share/clang-headers)
I still think there may be challenges with platforms like Windows where there's no Filesystem Hierarchy Standard to speak of, but this might be a way forward.
Does this sound like a way forward?
That sounds doable!
On the other hand I'm wondering whether iwyu should control this at all. There are only a few principle ways iwyu is being invoked. Through iwyu_tool, on a shell, or through CMake. Why not try to Do The Right Thing in these specific cases, but make it overridable through a resource-dir option?
Doing the right thing means guessing the resource directory. In particular:
- iwyu_tool: The compilation database contains compiler binary. From that we can determine the location of the headers.
- shell: Not sure what the right thing here is. We can try to guess the correct path by looking for usual suspect compilers.
- CMake: Here we'll need some help from CMake, which again knows the correct compiler. So it can supply a guess for the -resource-dir option
I know nothing about clang on windows however.
I think one challenge is that IWYU is tightly tied to a specific Clang version, namely the one it was built with. The builtin headers are revision-locked with the compiler source code.
So the compiler usually used to compile the user source, the one provided in compile_commands.json, is not necessarily the same version as IWYU's base compiler. Typically, it's not even Clang.
True, I didn't consider this. You can't really swap out intrinsic headers and expect things to go over smoothly. That leaves us only with shipping the headers. This makes packaging and distribution slightly more involved but not impossible. Something like $PREFIX/share/iwyu/include seems to be an appropriate location, at least for linuxoid systems.
Unfortunately the clang resource dir is not exported by clang's CMake. So again we can only make guesses. We could probably submit a patch to upstream though to export this.
@pfaffe posted a patch for the upstream exposure here: https://reviews.llvm.org/D49486. Thanks!
To come back to the potential solution, I think it comes down to:
- As a build step, copy the resource dir into the IWYU build tree:
$BUILD/share/iwyu/include - Change the current output path for
include-what-you-useto$BUILD/bin/include-what-you-use(rationale later) - Add a CMake install rule to copy them to the install tree at
$PREFIX/share/iwyu/include - Have IWYU set Clang's resource dir to
dirname(iwyu-binary)/../share/iwyu/include, which is pretty much what Clang does, I think
By building directly to bin/, it becomes possible to run IWYU directly from the build dir after build, especially if there's some kind of order dependency for copying the headers.
Does that sound about right?
As someone who's struggled off-and-on to provide usable IWYU packaging in Homebrew for several years (it's either "build the world" or "hackily symlink to Xcode's included headers and break during its upgrades"), if we get to a place where I can…
- Depend on
clangin Homebrew in a way it produces the header files for IWYU's use - Have IWYU put these in a well-known place (
shareis fine) - Let IWYU look for them here (possibly even override this with a
configureor CMake variable, so I can tweak it if things aren't entirely compatible?)
… then I will be very happy and packaging this in the future becomes much, much easier.
My plan was roughly this:
- Add a IWYU_RESOURCE_DIR cmake cache which defaults to CLANG_RESOURCE_DIR
- If they end up being unequal, install the entire CLANG_RESOURCE_DIR into IWYU_RESOURCE_DIR (This allows distinguishing behavior between building, installing and packaging!)
- Export IWYU_RESOURCE_DIR to the implementation, and plug it as a system header include path into the CompilerInstance
I'll cobble this together and paste a link!
Here it is: pfaffe/include-what-you-use@f4732dbeb2696266d60b97365630d4c099c47ba1
Currently this doesn't handle the difference between an install tree and a build tree correctly. This I'll have to figure out yet. If IWYU_RESOURCE_DIR manually set (e.g. to share/include-what-you-use), that directory doesn't mean anything in the build tree and becomes valid only after install. In this case we'd have to fall back to CLANG_RESOURCE_DIR. Not sure if cmake supports this nicely.
Cool! So IWYU_RESOURCE_DIR is always relative, then? That way the path gymnastics in iwyu.cc works out in the installed tree at runtime, which is a prerequisite for binary packages (e.g. for Windows).
Yeah, the build vs install tree was what I had planned to work around by brute-force copying the clang headers into the build tree if necessary, but that's pretty ugly. Or we could symlink them into the build tree (except that doesn't work on Windows, but I guess CMake falls back on copy).
It doesn't have to be relative. The default is equal to clang's CLANG_RESOURCE_DIR, which is most likely absolute. Is this an issue on windows?
Copying works, but it really is ugly. Maybe we can instead add both IWYU_RESOURCE_DIR and CLANG_RESOURCE_DIR as system include paths, but let IWYU take precedence. That way, if IWYU is set to something that points into the void (as it'd likely be in a build tree if it's set), we'd fall back to CLANG at runtime.
I can imagine it would be if there's a hard-coded absolute path compiled into IWYU. On Windows, the natural "packaging" paradigm is to build a GUI installer (often from a toolkit). The installer contains all files in an unrooted tree, sort of like a tar archive. The first thing a GUI installer typically prompts for is the install root (similar to the prefix, but Windows installers can write files anywhere they like on the disk). The default here would be something like C:\Program Files\include-what-you-use, but I might elect to install IWYU into F:\temp\include-what-you-use. So the install root can not be fixed at configure time, it needs to be resolved at runtime.
I'm not sure how stuff like this works in the Unix world -- I haven't had a chance to get acquainted with any packaging system -- can you not override the prefix for a package when installing it?
Also, why system include path rather than -resource-dir?
In Linux install behaves the same.
It didn't seem like i can properly pass the resource-dir into the tools. And there is only one resource dir, so the fallback thing wouldn't work in that case.
In Linux install behaves the same.
In that case I don't understand how hard-wiring an absolute path, decided at configure time, can work.
And there is only one resource dir, so the fallback thing wouldn't work in that case.
Ah, hm. I think it would make sense to make the fallback unnecessary, but yeah, until then, -isystem should be equivalent.
In that case I don't understand how hard-wiring an absolute path, decided at configure time, can work.
That only makes sense if at configure time the path of the clang resources is know. Otherwise, you can just specify a relative directory. The relative directory will then be relative to the install prefix.
OK, so if the IWYU package being built uses the same default install prefix as the user installing IWYU, this will work. And I guess that's the typical scenario. But in all other corner cases it will fail (e.g. user overriding install prefix, packager building with a non-standard prefix, user doesn't have clang installed, Windows [not really a corner case, but yeah :)]).
So it seems better to me to just always bundle the headers with IWYU.
Or maybe it's a build flag -- IWYU_BUNDLE_BUILTIN_HEADERS -- so that packagers can choose to bundle them, but lowly people like myself can just run IWYU from the build tree with the clang resource-dir path compiled in.
Let me clarfiy how I expected this to be used:
Use case a): If you build IWYU for yourself (i.e. you have the clang sources or at least headers around, and don't install or 'presist' the build anyhow), you just leave IWYU_RESOURCE_DIR alone, and everything will just work as you'd expect. At runtmie, iwyu picks up the headers that belong to the clang you used during build.
Use case b): If you build IWYU for distribution (i.e. packaging or local installation), you set IWYU_RESOURCE_DIR to a relative path. As result the resource headers will be copied into the bundle and picked up from there at runtime.
In my hacky implementation there are two caveats wrt. the uses cases:
- What if you don't want to bundle the headers but just have them in a weird location? You'd want to set IWYU_RESOURCE_DIR but not copy anything anywhere. This also is when you'd set the dir to an absolute path. A seperate BUNDLE option like you suggested would fix that.
- The divergence between build-tree and install-tree behavior we discussed above. This can be fixed by using the fallback onto CLANG_RESOURCE_DIR.
As noted in #827, the builtin headers are now copied to the build dir under all configurations. That solves this issue when include-what-you-use is executed from the build tree, which is not the case most of the time.
Next step should be to also install the headers together with IWYU. It's been quite some time since I investigated this, but I seem to recall there's no straightforward way to override Clang's idea of where the builtin headers should be. But there might be a more-or-less hacky way to do it.
I think there are a number of increments to this:
- Decide where the headers should be, relative to the IWYU install prefix. There's quite a bit to think about here, with side-by-side versioning, etc.
- Update IWYU to somehow prefer that location to the one it gets hard-wired from Clang (as described here: #100 (comment)).
- Update the build dir hackery to put them in the right place
- Have the install target put them in their prefix-relative home
- Provide enough build hooks and documentation for packagers to be able to customize
And they're all subtly interconnected. There are some additional quirks on macOS with Xcode to get include-what-you-use to use the right C++ standard library, but that's probably a nice challenge once all the other steps have been taken.
There's also some merit to the idea of having an IWYU package depend on clang for the relevant platform. But I think that might break down for Windows, where packaging isn't ubiquitous. And as far as I recall, our only runtime Clang dependency is the builtin headers, so it seems like a shame to require matching Clang version if we can bundle the headers anyway.
So, TL;DR; not sure what the next step is. Needs more investigation and thinking, which requires more time, of which I don't have much right now.