lbuild modules cannot acces lbuild repository names

[Nick Fiege]

Hi,

I have my own lbuild repository and in it are some lbuild modules.
My modules need to know the repository name which is then used to collect build-include paths and define output paths (which should be repo based). I do not want to hardcode the name of my repository (bad practice).

I cannot see any means to get the repository name from inside my module.lb files.
The only way I can see in code is like this:

    print(env.repopath())

which sadly returns the path of my module instead of the path of my repository.

@rleh suggested Queries, which can only find attributes of lbuild script by name ... but that defeats the point of being agnostic to the repo name

[Niklas Hauser]

Yeah, I've been meaning to add something like env.context which is a dictionary that every repo can fill with their own data. But then it conflicts conceptually with queries, so I didn't do that yet.

A hotfix would be to indeed use a repo environment query: env.query(":name") would simply return the string name. Note that you don't have to specify the repo name, since it's automatically resolved, so it would be agnostic from a module perspective.

[Niklas Hauser]

I mean I can certainly add a getter for the repo name to the module APIs, lemme try that in parallel.

[Nick Fiege]

Querie designs are almost never needed (in my opinion). These are usually a thing where magic meets magic. In this case, a environment dict with the current context would be the first thing I would have implemented. Queries should be used to query external information from other modules (not current environment).

In my opinion: Modules should be agnostic to other modules or their repositories by design.

[Niklas Hauser]

Querie designs are almost never needed

modm uses queries to modularize functionality:

 $ ack -C 1 add_query
tools/build_script_generator/module.lb
97-    # Queries
98:    module.add_query(
99-        EnvironmentQuery(name="source_files", factory=common_source_files))
100:    module.add_query(
101-        EnvironmentQuery(name="device", factory=common_target))
102:    module.add_query(
103-        EnvironmentQuery(name="memories", factory=common_memories))
104:    module.add_query(
105-        EnvironmentQuery(name="avrdude_options", factory=common_avrdude_options))
106:    module.add_query(
107-        Query(name="collect_flags", function=common_collect_flags_for_scope))

ext/st/module.lb
151-
152:    module.add_query(
153-        EnvironmentQuery(name="rcc-map", factory=common_rcc_map))
154:    module.add_query(
155-        EnvironmentQuery(name="headers", factory=common_header_file))

src/modm/platform/core/cortex/module.lb
247-
248:    module.add_query(
249-        EnvironmentQuery(name="vector_table", factory=common_vector_table))
250:    module.add_query(
251-        EnvironmentQuery(name="linkerscript", factory=common_linkerscript))

src/modm/platform/usb/stm32/module.lb
96-
97:    module.add_query(EnvironmentQuery(name="irqs", factory=common_usb_irqs))
98-

In this case, a environment dict with the current context would be the first thing I would have implemented

Queries are a global context for sharing data and code, except they are documented and discoverable via lbuild discover --developer.

For example in modm:

 $ lbuild discover ::memories
>> modm:build:memories  [EnvironmentQuery]

Extracts the memory map of the device.
A memory region is a dictionary containing:

  - `name` of region
  - `start` address of region
  - `size` of region
  - `access` of region

:returns: a list of memory regions.

Please try if env.query(":name") works for you.

[Nick Fiege]

Yes I will try it :)

It has to wait until Monday I guess :D

With never needed i meant that the purpose of queries (as you seem to use it) are just dictionary lookups of contexts between hierarchical elements (repo/module). It just seems weird to me that there would be a use case for module to module (horizontal connection in a hierarchical tree) communication of data. (Correct me if my idea is wrong)

I'm just a little bit confused on why such a hard structure (maybe my view on this is a little odd) was chosen for simple dict lookups and type casts (especially in python).

These are all just "attributes" of modules and attributes can be looked up or extracted from them, the word query has so much more weight and it doesn't seem "reasonable" to me (whatever that is worth :D)

No offense, I am just trying to get a grasp on some of the design decisions <3

[Niklas Hauser]

lbuild and modm are about modularization of responsibilities. There simply are no trivial centralized solutions when generating a HAL for 3000+ different devices.

We cannot have a giant repo.lb file that computes everything globally, instead need to compute the relevant data in the module that has expertise on this (good locality of data/code). Queries are a way to removing duplicate code by implementing an repo-internal API of shared code/data and having the choice to implement it where it makes the most sense, especially without having to know the file paths or options. That way refactoring and maintenance gets easier, particularly in such a large project.

If you really need to access complex data in your module, then you can use all of the Python import mechanism to do so.
For example the modm:build module imports a common.py file manually to share some common code and data with the other build modules. You have a full Python script in every module.lb, lbuild just happens to calls some functions in it ;-p

Also keep in mind that the module tree is not static, it very much changes depending on :target option. Therefore you have have subtle differences in computation that only matter to the module at hand. The queries give you an abstractions.

This also means that lbuild is not necessarily that useful for projects that don't have much to configure. Outside of modm/platform and modm/ext there isn't that much need for code generation, instead the needs for a traditional package manager becomes stronger, which basically also "just" resolves dependencies and copies files and build scripts around.

[Nick Fiege]

I played around with the queries and this worked out for me:

repo.lb

def getRepoName(env={}):
    return "erpc_modm"

def init(repo):
    repo.name = getRepoName()
    repo.description = FileReader("README.md") 
    repo.add_query(Query(name="repoName", function=getRepoName))

module.lb

def build(env):
    repoName = env.query(":repoName")()

Although i think that this should be in the env auomagically, this solution works and i can sleep better now! :)
Thanks for the insides and the help <3

[Niklas Hauser]

If you use an EnvironmentQuery, you don't need to call the function.

repo.add_query(EnvironmentQuery(name="repoName", function=getRepoName))

repoName = env.query(":repoName")

[Nick Fiege]

repo.add_query(EnvironmentQuery(name="repoName", function=getRepoName))
TypeError: init() got an unexpected keyword argument 'function'

seems to me that there is some more kwargs** handling needed! :D

[Niklas Hauser]

I guess I called it factory to distinguish it from a callable function, since it's constructing data only.

repo.add_query(EnvironmentQuery(name="repoName", factory=getRepoName))

[Nick Fiege]

that works, Im using that now :)
thanks