lbuild does not work properly when modm is included relative to module

[Nick Fiege]

HI,

I am trying to set up a modm module, which keeps modm internally (as a submodule; for versioning purposes). Sadly lbuild seems to have some bad manners regarding that use case:

including modm into my project and adding its .lb file to my project.xml like this

  <repositories>
   <repository>
      <path>./external/modm/repo.lb</path>
    </repository>
  </repositories>

leads to two instances of modm when i invoke scons (which will fail because modm now exists twice).

• Is it possible to tell the build system to ignore files? I can't see anything in the lbuild documentation. I would probably ignore the modm (original) files, adding external (which is the dir where modm is) to the SConstruct script file likes this ignored = [".lbuild_cache", env["CONFIG_BUILD_BASE"], "external/*"] + generated_paths works, but is of course not a good solution.
• After all, the other problems below are still the main blockade here ...

My second approach was to only add my own module.lb file for my repository and adding modm in there like this:

  <repositories>
    <repository>
      <path>./module.lb</path>
    </repository>
  </repositories>

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

def init(repo):
    repo.name = "low-level-compute"
    repo.description = FileReader("README.md")
    return repo 

def prepare(repo, options):
    repo.add_modules_recursive("external", modulefile="*.lb")
    return True

def build(env):
    env.collect(":build:path.include", "include")

In this instance, lbuild build fails with

ERROR: In '<lbuild.module.ModuleInit object at 0x7f85e6a7ec10>':
Traceback (most recent call last):
  File "/low-level-compute/external/modm/repo.lb", line 213, in init
    repo.add_ignore_patterns("*/*.lb", "*/board.xml")
AttributeError: 'ModuleInitFacade' object has no attribute 'add_ignore_patterns'

inside modm itself.

My third attempt was to not include modm at all and just use my own module.lb, but lbuild doesnt like that at all either:

ERROR: Repository(low-level-compute) in 'module.lb' did not add any modules!

How can lbuild be that inflexible? I thought that lbuild modules can be used recursively (up and down). It should'nt matter that a module does'nt include other modules ... modm should not crash when beeing imported (NO, lbuild should work without modm beeing sourced first).

project.xml commands like <extends>modm:nucleo-f767zi</extends> are parsed first without loading modules, which means that whenever someone does not include modm as the first thing inside his project.xml file, every other option below will fail.

What is happening here? Is my use case so out of touch? It seems to me that lbuild is way to restricted and not really recursively packaging at all.

[Niklas Hauser]

I am trying to set up a modm module, which keeps modm internally (as a submodule; for versioning purposes)
What is happening here? Is my use case so out of touch?

I'm a little confused too. The recommended way for versioning is to use a git submodule, which maintains it's SHA version, in an git_repo/ext/modm folder, which is separate from the git_repo/app/project_name/project.xml folder, and including the modm lbuild repo path relative: ../../ext/modm/repo.lb. I also recommend committing the generated sources so that when you upgrade modm via lbuild build you can see the source changes. Committing the generated sources also removes the need to deal with lbuild for other team members.

leads to two instances of modm when i invoke scons

All build systems recursively search for buildable files, therefore it will build everything inside the modm git repo as well.
I very strongly recommend placing modm outside of your app folder.

My second approach was to only add my own module.lb file for my repository and adding modm in there like this

repo.lb != module.lb and the root repo node is special and must be parsed first before all modules, therefore you cannot include it as a module in another repo. You can however include as many (unique) repos as you want.

All modules inside modm depend on the repo option modm:target (which contains the modm-devices data) and evaluate it first inside their own prepare stage to decide availability. For example there are four modm:platform:core modules (avr, stm32, sam, hosted), but only one makes itself available depending on the chosen modm:target platform.

The repo.lb also defines things like configurations, so that <extends>modm:nucleo-f767zi</extends> can be resolved first as you discovered. There must also be at least one module.lb available, otherwise name resolution will get wonky.

How can lbuild be that inflexible?

lbuild is mostly a modular code generator, any similarity with a package manager is purely accidental 🤪…
To be specific, lbuild competes with kconfig as a user-facing configuration API with code generation capabilities.

It seems to me that lbuild is way to restricted and not really recursively packaging at all.

It's still not clear to me what your use-case is? What additional versioning/packaging do you need that a git submodule cannot give you?

[Nick Fiege]

which maintains it's SHA version, in an git_repo/ext/modm folder, which is separate from the git_repo/app/project_name/project.xml folder, and including the modm lbuild repo path relative: ../../ext/modm/repo.lb. I also recommend committing the generated sources so that when you upgrade modm via lbuild build you can see the source changes. Committing the generated sources also removes the need to deal with lbuild for other team members.

All build systems recursively search for buildable files, therefore it will build everything inside the modm git repo as well.
I very strongly recommend placing modm outside of your app folder.

Well the submodule part is irrelevant, it already is a submodule so thats fine I guess. I will try to implement your recommendation by putting modm ABOVE the application structure (this is not really intuitive and also a little bit messy but if it works, thats fine for me)

repo.lb != module.lb and the root repo node is special and must be parsed first before all modules, therefore you cannot include it as a module in another repo. You can however include as many (unique) repos as you want.

what ?? Whats the difference? How can someone know that? Both look equivalent ... how does lbuild differentiate between them? by Name? Thats a little backwards don't you think? Why not make that clear via the API? As to why there should be a difference at all is a mystery to me but these could at least be seperated by design ... not filenames or some hidden mechanism.

It's still not clear to me what your use-case is? What additional versioning/packaging do you need that a git submodule cannot give you?

Well if everything above is considered (and used correctly) it should be fine.

lbuild is mostly a modular code generator, any similarity with a package manager is purely accidental zany_face…

That is an easy way out, but unfortunately a lazy approach to this problem (I am fine with lazy approaches). lbuild in my opinion (I want to make it clear that I am not fully grasping the potential of lbuild yet) should be more than that and in some cases already is more. lbuild utilizes mechanism's for packaging like

• project definitions
• modules
• source code provision
• build tool preparations
• code generation templates
• etc.

Well that's just philosophy, I am not here to bash design decisions ... I just want to understand. So please dont take my comment in an offensive way :)

[Niklas Hauser]

what ?? Whats the difference?

The API and execution order.

How can someone know that? Both look equivalent... Why not make that clear via the API?

True, they look similiar, however, the docs are fairly clear on the difference.
The modm guide also makes a deal out of repo files: https://modm.io/guide/discover/

how does lbuild differentiate between them? by Name?

lbuild doesn't distinguishes at all, the repo.lb and module.lb are just convention. You can call your files hello_world.exe for all lbuild cares, therefore you need to pass the full path to the repository file in your project.xml, not just a directory. If you look into the modm/driver folder, you'll see that most module.lb files are renamed to the driver_name.lb to fit more than one module into a directory.

lbuild in my opinion should be more than that

I think you're conflating lbuild with lbuild+modm. lbuild only does code generation and modular configuration management.
It's just dumb plumbing, all the packaging stuff is done by modm:build and it's submodules, while the data and HAL management is done by the modm:target option via modm-devices. Note that in the end your actually using modm+SCons with lbuild completely out of the picture (hence our recommendation to commit the generated code).

That is an easy way out, but unfortunately a lazy approach to this problem

I feel you… however, lbuild (or modm) cannot compete with package managers and embedded libs written by hundreds of full time employees. So we pick our battles, and lbuild has turned out to be pretty good at enabling a much better HAL design in modm.

[Nick Fiege]

your tips were are a helpful clarification. Although I am not really pleased with my application structure, it works for now.

• myApplication
  • libs
    • modm
    • other third party libs and mod repositories / modules and stuff
  • app
    • include
    • src
    • project.xml

If i could suggest a use case for lbuild, it would be the possibility that modm (or any lbuild repository) could be inside it's own directory inside the application and that project.xml could recursively add repositories from a directory.

Something like that:

• myApplication
  • libs
    • modm
    • other third party libs and mod repositories / modules and stuff
  • include
  • src
  • project.xml

with a project.xml containing this

 <repositories>
   <directory>
     <path>./libs/</path>
   </directory>
 </repositories>

[Niklas Hauser]

If i could suggest a use case for lbuild, it would be the possibility that modm (or any lbuild repository) could be inside it's own directory inside the application

In our experience, we often have more than one application per project, usually a few small ones that test only one small part of the hardware in isolation for a controlled bring-up, and then a full featured firmware. Therefore a shared modm repo outside of the application folder is useful. It also makes it easier for IDEs to only see the generated code, not the whole repo.

You can even use an environmental variable for resolving the repo path.

project.xml could recursively add repositories from a directory.

I think you'll survive adding the full path to modm/repo.lb 😜 At least until there more than one lbuild repo on this planet.

[Nick Fiege]

Well the problem is that multiple applications inside our workspace (multiple firmwares) could depend on different versions of modm in the future. I wouldn't hope that this is the case but we cannot be sure that modm keeps its compatibility for the next 2/4 years. At some point, modm will change and design decisions (as well as external factors on our side) will render some code unmaintained. That's why having modm (as well as other libs) for each separate building block is useful (and also redundant, but that's a submodule problem).

I can live with that structure. But it is slightly annoying to deal with, but that's just personal preference I guess. The thing I don't like here is not that the repo structure is annoying, it is that modm enforced it onto us. That's why I formally announced my use case because I think that modm (+ lbuild) should try to not enforce anything onto the user that is not strictly necessary :)

[Niklas Hauser]

Now that's an interesting use case!

My recommendation is to commit the generated sources for each application. Then you can

1. modify them in place for experimentation.
2. update them individually for each app.
3. see directly what changed inside the generated sources when you do an lbuild build in the future.

[Nick Fiege]

I know about that (because it is written in the docs) and @rleh has already talked about it ... but I don't like this approach. Maybe my opinion on that is wrong ... but I don't feel comfortable with giving modm that kind of special position inside the software. It seems special to me, because:

• modm then becomes part of the use case / application code instead of something external
• modm get's priority handling (and versioning) in respect to every other code library
  • instead of a 'not under version control' build artifact (as it is right now)

As I said, my opinion could be wrong .. so will talk with the other devs in our team about this :)
But I'd rather take a directory separation than special pleading for modm :)

Although the fact that i have to clone modm 15 times (separate for every firmware) now is slightly annoying (submodules and stuff)

[Niklas Hauser]

Ok, then I'm closing this issue and later if you want to report back what solution you've come up with, we can derive new requirements from that in a new issue.

[Nick Fiege]

Thanks for the help :)