west init: CLI argument to automatically setup new workspace from template
jamie-bes opened this issue · comments
There are three basic types of Zephyr applications: Zephyr repository applications, workspace applications, and freestanding applications.
The default behavior of west init
without any arguments effectively creates a Zephyr repository application, as upstream Zephyr becomes the manifest repository. This is nice for building the samples included in the Zephyr source tree, but creating a new application within this structure typically requires forking Zephyr. The workspace application structure is often better for developing new applications, as the repo for your application only needs to contain the application itself while Zephyr gets pulled in as a module. But the workflow for creating a workspace application is a bit more verbose, as you need to provide an already-existing manifest repository with a west.yml
file and pass either its URL (with -m
) or a local file path (with -l
) to West.
My proposal is to add a new argument that will automatically create a template manifest repository with west.yml
and other required files, such that it can immediately be used as a workspace application. The idea comes from Rust's cargo init
, which automatically initializes a new Git repo with Cargo.toml
, Cargo.lock
, .gitignore
and a src/main.rs
with a basic "Hello World" function. This makes starting a new application extremely easy, especially for beginners, so I think it'd be a nice feature to replicate in West.
I think west init --template
or west init -t
for short would be a decent name, though I'm open to other ideas. Since this probably couldn't be used in combination with -m
and --mr
due to the manifest repository being created from scratch, it'd be handy to have additional arguments to specify which version of Zephyr to include in west.yml
(maybe something like west init -t --base https://github.com/nrfconnect/sdk-nrf
and west init -t --br v3.3.0
).
As for what should be included in the template, I'd recommend this:
src/
├─ main.c
.gitignore
CMakeLists.txt
prj.conf
west.yml
...where .gitignore
ignores the build
directory, west.yml
imports the selected base
(defaulting to https://github.com/zephyrproject-rtos/zephyr) and nothing else, and the remaining files mimic the contents of the Hello World sample. I could also see an argument for using https://github.com/zephyrproject-rtos/example-application as the template, though it's a fairly maximalist template and provides more than the average user needs. So restricting the template to the bare minimum needed to run "Hello World" makes more sense to me.
If it helps, I'd be happy to put together a PR for this feature, though I created this issue to get some feedback on the idea first.
it'd be handy to have additional arguments to specify which version of Zephyr to include in west.yml (maybe something like west init -t --base https://github.com/nrfconnect/sdk-nrf
This template location argument should not be just "handy" but required and it should be designed ready to support projects that don't use Zephyr at all. I know the location of the west documentation makes that confusing but west
is actually a tool independent of Zephyr and we want west
to become more and more independent from Zephyr:
From #408 (comment)
We'd like to solve these types of problems in general by divorcing zephyr base setting from west itself,
A bare west init
command defaults to Zephyr and that can't be changed for obvious backwards compatibility reasons but that should be the exception not the rule. Other, newer options should require a Zephyr-specific parameter/URL
EDIT: this of course means most of the template details should live in the Zephyr repo, not in the west repo.
That caveat aside, the idea sounds good to me.
If it helps, I'd be happy to put together a PR for this feature, though I created this issue to get some feedback on the idea first.
Indeed! Thanks for the offer, much appreciated.
But please wait until @mbolivar-nordic / @mbolivar is back from vacation, I would hate you to write some code only to hear that it should be done but in a totally different way that he already had in his head for some time or even as a prototype in some branch.
Ah I see - I wasn't aware of the efforts to make west
more independent from Zephyr, so my proposal was pretty Zephyr-specific. I can try to generalize here.
This template location argument should not be just "handy" but required and it should be designed ready to support projects that don't use Zephyr at all.
A bare west init command defaults to Zephyr and that can't be changed for obvious backwards compatibility reasons but that should be the exception not the rule. Other, newer options should require a Zephyr-specific parameter/URL
I hadn't intended --base
as a template location argument, but rather an override for whatever default imports the template has in its west.yml
(eg. for Zephyr, overriding importing upstream Zephyr with a fork like NCS). So I considered it a nice-to-have but not strictly necessary as the user could just modify west.yml
manually.
But if we want to explicitly specify what template to use (rather than defaulting to anything Zephyr-specific), we could add that as a required argument to --template
. Using the URL of the template wouldn't have much value, as it would just be equivalent to west init -m https://github.com/zephyrproject-rtos/example-application
-- the convenience of this proposal really comes from not needing to figure out what URL to use. But something like west init --template zephyr
(shorthand west init -t zephyr
) could work really well, where the zephyr
template isn't built in to West itself but gets installed into some local directory of templates. And then if some other platform XYZ is also using West, a user could install a template for XYZ and be able to run west init -t xyz
, without the West repo needing to have any implementation details specific to XYZ.
In #261, @mbolivar-nordic mentioned the idea of deprecating the bare west init
that defaults to Zephyr.
We could similarly deprecate this slowly by emitting a warning in west init if -m is not given, and eventually make it a required argument over a period of a few years.
And for Zephyr users, I think west init -t zephyr
could actually become the de-facto replacement for bare west init
, providing the extra convenience of setting up a template "Hello World" app and being less verbose than west init -m https://github.com/zephyrproject-rtos/zephyr
.
But please wait until @mbolivar-nordic / @mbolivar is back from vacation, I would hate you to write some code only to hear that it should be done but in a totally different way that he already had in his head for some time or even as a prototype in some branch.
Absolutely, no problem!
I don't know about you specifically but there's one thing I'm afraid a lot of people don't realize[*]: when they run west build
, none of the code that runs is actually part of west
. It all comes from https://github.com/zephyrproject-rtos/zephyr/tree/main/scripts/west_commands
[*] because they don't care, and that's OK.
I see that @mbolivar / @mbolivar-ampere is back now - what are your thoughts on the idea of being able to install workspace templates and initialize them with west init -t <template_name>
?
The general idea seems like a great way to save people time. I'd like to know more about how the "local directory of templates" should get set up and maintained and how the namespace will be controlled, though. Naively I too thought "this should be a URL", but I could easily be convinced otherwise with some more details about the ergonomics of that local directory!
There's a lot of different ways that mapping between a name and a template could be handled. A package manager with a universal registry and controlled namespacing would likely just use some convention on top of their existing package namespacing scheme (like what Yarn does with yarn create
and packages prefixed with create-
).
But for West, I think the most logical approach is to just use a West workspace, since west.yml
already handles assigning project names to repository URLs. The flow could look something like this:
- An install script for West (either
setup.py
or something invoked by it) would do the following:- Initialize a new workspace in
$HOME/.west_templates
with an emptywest.yml
. - Run a command like
west config --global templates.path $HOME/.west_templates
to set West to use that workspace for looking up templates.
- Initialize a new workspace in
- When installing Zephyr (or any other framework that uses West), their installation guide could optionally recommend any relevant template repositories and give an example snippet of what the user should copy into their
$HOME/.west_templates/west.yml
to "install" them. - Running
west init -t my_template
would look for a project namedmy_template
in$HOME/.west_templates/west.yml
and use that as the manifest repository to initialize a new workspace (which could just callwest init -m <template_url> -mr <template_revision>
under the hood).
Note that this flow doesn't actually run west update
in the templates workspace -- it's just pulling info from its west.yml
. But I could imagine another approach that does use west update
and uses the cloned repository contents as a cache of the templates so they don't have to be downloaded every time you run west init -t <template_name>
.
I'd like to know more about how the "local directory of templates" should get set up and maintained and how the namespace will be controlled, though.
Very important question indeed.
And for Zephyr users, I think
west init -t zephyr
could actually become the de-facto replacement for bare west init, providing the extra convenience of setting up a template "Hello World" app and being less verbose thanwest init -m https://github.com/zephyrproject-rtos/zephyr
.
I like this. You could also extend it to west init -t zephyr/some_advanced_sample
where west init -t zephyr
is a shortcut for west init -t zephyr/default
(similar to /index.html
, you get the idea).
But for West, I think the most logical approach is to just use a West workspace, since
west.yml
already handles assigning project names to repository URLs. [...]
Note that this flow doesn't actually run west update in the templates workspace -- it's just pulling info from its west.yml.
Recursion is always sexy but I'm not convinced using west
to manage a "registry" of templates brings a lot of value, it feels overkill to me. I mean you don't need a tool like west
to implement a basic dictionary/map. Or rather, it feels premature: an implementation detail that should not taint the user interface anyway and that can be decided later.
For any registry like this I think the critical design decisions that must be right the first time (cause they're super hard to change later) are:
- how many layers of indirections
- how many "repositories" (typically: git repos but not necessarily)
- who has the authority (and the duty) to add/update/delete the template structure.
The hosting, download and caching technologies can be figured out later, maybe even enriched/diversified long after the first release.
It should always be possible to override from the default locations using some "long" arguments with full URLs (e.g. --registry http://myserver/myregistry
or --template file:///home/me/mytemplate
) but I understand that most users should not have to enter any URL; that's an important part of the desired feature
I think the 1st, "registry" level should be hosted and owned by https://github.com/zephyrproject-rtos/west (what else? (c)). It would do this sort of mapping:
zephyr -> https://github.com/zephyrproject-rtos/zephyr/ templates
# that's all for now! But enough for `west` not to be zephyr-specific.
The 2nd, actual template level should be hosted and owned by https://github.com/zephyrproject-rtos/west
template1 -> templates/templates1
"" -> templates/default
So in the end:
west init -t zephyr
=> https://github.com/zephyrproject-rtos/zephyr/ templates/default
west init -t zephyr/template1
=> https://github.com/zephyrproject-rtos/zephyr/ templates/template1
Thoughts?
In a totally different issue I just mentioned https://docs.platformio.org/en/latest/frameworks/zephyr.html.
Maybe they have good ideas on this topic.
@marc-hb The namespacing structure you described makes sense to me, though we might be starting to discuss two different (though interrelated) features here.
- The
west init -t ...
command syntax, which would be a fairly simple wrapper aroundwest init -m ...
but would run an extra lookup step at the beginning to find what URL to use for the manifest repository rather than having the user pass it directly.
If the namespace for the lookup is strictly local (eg. a particular user has zephyr
mapped to some official template in their local config but another user might choose to map zephyr
to something totally different), then this would be a pretty small, self-contained feature, with the only additional design work being deciding how to structure the namespace configuration files (maybe using west.yml
as I described earlier or maybe something simpler like an INI file). But it would require the user to maintain their own namespace configuration. However, if we wanted west init -t ...
to use the same namespace for all users and work without any additional configuration from them, we'd also need the following feature:
- A hosted registry. If this were to exist,
west init -t ...
could do a remote lookup to find the URL for a particular template name similar to what I alluded to withyarn create
. But if we decide to do a hosted registry (or even if we do local config in the short term but plan to move to a hosted registry in the long term), controlling the namespace and preventing name conflicts becomes a bigger concern. Sorting out the design issues there and implementing the registry itself would be a challenge, which is why I was suggesting the local config option earlier, but if there's interest in a hosted registry, it would likely result in a better user experience forwest init -t ...
. And outside of the context of project templates, a general West project registry would be extremely useful. Imagine you're working on a Zephyr application and need to use cryptography functions from Libsodium, and you could runwest add zephyr/libsodium
thenwest update
to automatically add it to yourwest.yml
and install it.
I created this issue with only feature 1 in mind, but feature 2 is pretty compelling as well and might inform our design for feature 1. But I think designing feature 2 properly would require a lot of effort and would get pretty heavy into "make West a universal package manager" territory, so if it's something that we're interested in, we should probably make a separate Github issue for it.
A hosted registry. If this were to exist, west init -t ... could do a remote lookup to find the URL for a particular template name similar to what I alluded to with yarn create.
But if you don't provide a default registry (which again, should be very easy to override on the command line), then you must provide instructions to download a "local" registry by following a... URL, so you just moved the URL problem one step further! Haven't you?
But if we decide to do a hosted registry (or even if we do local config in the short term but plan to move to a hosted registry in the long term), controlling the namespace and preventing name conflicts becomes a bigger concern.
Your scalability concerns feel even more ambitious than mine considering west
has only one high-profile and well known project using it right now but... I'll roll with them! So how about org.zephyrproject -> https://github.com/zephyrproject-rtos/zephyr/ templates
. Very simple and proven; namespace problem solved.
I created this issue with only feature 1 in mind, but feature 2 is pretty compelling as well and might inform our design for feature 1.
Yes, that's why I mentioned it. I agree you can start coding and testing with only a local registry as long as you keep the design open for a global one later.
@jamie-bes please take a look at
It's not directly related but it's in the same "area".