Add symlinking ability to lbuild
ASMfreaK opened this issue · comments
I'm developing library.
I'd like to have ability to symlink files instead of copying for debug. Something in lines of:
def build(env):
env.outbasepath = "some/dir/to/output"
env.link("some_file.hpp")I can achieve this behaviour using the following script, but it's way too clunky:
def build(env):
env.outbasepath = "some/dir/to/output"
for fn in ["some_file.hpp"]:
ofn = env.real_outpath(fn)
env.copy(fn)
os.unlink(ofn)
os.symlink(env.localpath(fn), ofn)
Yeah, this came up a few times already, there's already a branch for that: https://github.com/modm-io/lbuild/tree/feature/symlink
This allows lbuild build --symlink, which symlinks every env.copy, but not env.template or env.archive for obvious reasons.
I hesitate to merge this, because it trains people to change the generated files. That's fine as long as you know which files are copied/linked and which are generated. The time will come when you make changes to a generated file and it gets overwritten.
Is there a use-case for exposing symlinking via env.link in lbuild? The issue is that it creates a dependency on the original lbuild repo to be available, which isn't something we want.
The time will come when you make changes to a generated file and it gets overwritten.
Hm, on the other hand, there are a few issues open on tracking the content of the generated files (with hashing) and to generate warnings about what files were modified and are about to be overwritten: like #33.
With that I would be more comfortable to add this features.
Let me know if this makes your life easier, and i'll just merge it with a big fat warning.
You can install this branch via:
pip3 uninstall lbuild
pip3 install git+https://github.com/modm-io/lbuild@feature/symlink
Is this possible on per-module basis?
Something in lines of:
def init(module):
# ...
module.symlink_without_copy = True
# ...This is false by default (if not with --symlink)
Hm, definitely not a fan of changing the module API, I was thinking more of adding module names via lbuild build --symlink repo:module1 --symlink repo:module2:submodule etc, like lbuild build -m repo:module works today.
Can you explain the use-case for this? Is there an issue with symlinking all modules? Why just a few modules?
I thought of opt-in approach for debug purposes only.
I can't think of any issues with symlinking all modules.
I tried enabling symlinking in my modm project.
Well, it didn't went too well:
Compiling C++·· build/release/modm/src/modm/io/iostream.o
<REDACTED>/modm/src/modm/io/iostream.cpp:16:10: fatal error: iostream.hpp: No such file or directory
16 | #include "iostream.hpp"
| ^~~~~~~~~~~~~~
compilation terminated.
Is this an issue with using symlinks with GCC on Windows/NTFS?
Mixing copied and templated code does not work.
I'm using Arch Linux, arm g++ 9.2.0.
If I understand correctly, compiler is deducing current directory of the original file, but not symlink. It fails to find templated code in the source directory.
I can reproduce this, why is GCC following symlinks now? I'm still on ARM GCC v8.3.1. Is there a flag we can pass to GCC?
Ok, so it follows the symlink to the real file, which is in the modm repository, but of course the include file isn't there because it's a template iostream.hpp.in. Unless there is a flag to turn this behavior off, this feature cannot work at all.
So… this issue is dead, right? Or did you find any ways that you can make this work for GCC?
Wait. GCC is not the culprit here. It's actually lbuild.
This is what lbuild generated in modm/SConscipt:
files = [
env.File(r"../../../modm/src/modm/architecture/interface/can.cpp"),
env.File(r"../../../modm/src/modm/architecture/interface/can_message.cpp"),
env.File(r"../../../modm/src/modm/debug/error_report.cpp"),
env.File(r"../../../modm/src/modm/debug/logger/hosted/default_style.cpp"),
env.File(r"../../../modm/src/modm/driver/io/terminal.cpp"),
env.File(r"../../../modm/src/modm/io/iostream.cpp"),
env.File(r"src/modm/io/iostream_printf.cpp"),
env.File(r"../../../modm/src/modm/math/utils/bit_operation.cpp"),
env.File(r"../../../modm/src/modm/math/utils/pc/operator.cpp"),
env.File(r"src/modm/platform/core/assert.cpp"),
env.File(r"src/modm/platform/core/clock.cpp"),
env.File(r"../../../modm/src/modm/platform/core/hosted/memory.cpp"),
env.File(r"../../../modm/src/modm/utils/dummy.cpp"),
]The ../../../ - is where the actual modm source repo is located.
No wonder I got those include errors.
What can be different from your setup?
I have:
$ lbuild --version
lbuild 1.12.4
$ python --version
Python 3.7.4
And this is the real culprit (os.path.realpath):
def reloutpath(self, path, relative=None):
# ....
return os.path.relpath(os.path.realpath(path), os.path.realpath(relative))which is called from here in modm:
env.template("resources/SConscript.in", "SConscript",
filters={"flags_format": flags_format,
"relocate": lambda p: env.relative_outpath(p, repo)})which was called from template:
env.File(r"{{ file | relocate | modm.windowsify(escape_level=0) }}"),So symlinks are actually a viable solution, if this issue with relocate can be somehow resolved in lbuild or modm
Oh, that's bad! I don't know why I used realpath there, this may not be intended behavior.
I fixed that issue in the lbuild branch and this fixes compilation for me! So I'll open a PR with the feature.