Modules stabilization tracking issue
casey opened this issue · comments
This issue tracks stabilization of mod statements.
There are a bunch of features which aren't yet implemented, but I think they can all be implemented in a backwards compatible fashion:
- Recipes defined in one module can be called from another
- Variables defined in one module can be referenced from another
- Allow inline modules that are declared and defined in a single file.
- Allow submodules to opt-in to using parent module settings
- Allow submodules to opt-in to loading
.envfiles - #2119
I want the feature to be well used and well tested before stabilizing it.
Hi @casey, curious if this project is inspired to any extent by mmake? For me, the most painful thing that traditional make lacks is support for remote includes.
For example, we have built out a set of common makefiles at my job for common projects (e.g. ExpressJS/Node apps, Spring Boot apps, AWS Lambda with SAM CLI, etc.). We also have a core makefile that exports handy commands to print messages in certain colors, require certain environment variables, etc.
The most annoying part of this setup is that we have to ask people to copy/paste files into their repos. We even have an update-makefiles command so that once they do so, they can consume updates without friction.
I just wanted to throw this out there. I know there is a lot of thought being put into a module system in general, but allowing for remote modules would be awesome.
Thanks for all the hard work on a really cool tool :)
Hi @casey, curious if this project is inspired to any extent by mmake?
I didn't know about mmake but it seems to be in a similar vein, although just pre-dates it by a couple years. In fact, just also used to be a wrapper-script around make. I've been meaning to add a list of alternative command runners to the readme for a while, so I just opened #1008 which mentions mmake.
For me, the most painful thing that traditional make lacks is support for remote includes.
That's a super interesting feature. We don't have modules, but if we ever do, I wouldn't be opposed to adding something like that.
For example, we have built out a set of common makefiles at my job for common projects (e.g. ExpressJS/Node apps, Spring Boot apps, AWS Lambda with SAM CLI, etc.). We also have a core makefile that exports handy commands to print messages in certain colors, require certain environment variables, etc.
Interesting, that does seem very useful.
I just wanted to throw this out there. I know there is a lot of thought being put into a module system in general, but allowing for remote modules would be awesome.
Feel free to open an issue about remote modules, why they're cool, how you use them. It's something that won't get implemented for a long time, but having an issue for it would probably be good.
Thanks for all the hard work on a really cool tool :)
You are most welcome! It is good to be of service 🙌
Found this issue also hoping it was talking about remote includes. We currently use make, and rely heavily on a centralized makefile. It's distributed using an include line that uses a shell command to invoke curl:
include $(shell test -f .common.mk || curl -sSL -o .common.mk http://url/to/common.mk; echo .common.mk)
Works pretty well, but some of the frustrations around make have us always looking for alternatives. Which is how I found just, and now I am evaluating whether it will work for us...
I would also jump in with support for remote modules.
One of the hardest things about working with cloud infrastructure and deploying applications is the need for sharing and reusing.
Wanting to reuse all these great just commands I've written in one repository are not easily transferred to another repository.
I feel like deno has solved this problem (URL imports, versioned, immutable repos), and aim to write all my future cloud related coordination and scripting.
But it doesn't get around not being able to re-use just commands where I really want them.
I think this tool really appeals to those who pipeline software.
I'd love to have something analogous to url imports in dhall for importing modules in just. A simple solution without caching or checksums would be enough for my needs. I'd likely use commit hashs/tags for versioning and whether or not a 50ms http request is cached or not doesn't matter to me.
Fetching from private repos is probably a more important feature and requires some though in the general case, for github nothing special is required as one can just put the personal access token in the url.
Importing from public repo:
keybase := import https://raw.githubusercontent.com/casey/just/v0.9.4/examples/keybase.just
Importing from private repo
foo := import https://{{TOKEN}}@raw.githubusercontent.com/some_org/some_private_repo/v0.1/foo.just
where TOKEN is either supplied via the command line or via environment variable.
Thanks a lot for the new module system in v1.19.0 🔥
It'd be amazing to have the following feature (you already envisioned) in subsequent releases:
Allow specifying the path to the module source file, so a module justfile can be anywhere.
This is since I'd like to have one commands/ folder where I put in all my submodule .justfiles, e.g. docker.justfile, test.justfile etc. and have one project root .justfile. Having to create a new folder for every of them only clutters the workspace. I feel like how you have your project structured in folders does not necessarily correlate to how you want your commands to be modularized.
Here is the way I achieved this behavior so far.
Thank you @casey for hanging in there on the import and module features!
Nice, this looks like it could be an elegant replacement for justfile recipes such as
[no-exit-message]
foo +args:
{{quote(just_executable())}} -f some_repo/foo/justfile {{args}}
the purpose of which is to run just foo recipe to run a repository's just recipes with some local custom environment settings.
But in testing out replacing this recipe with mod foo 'some_repo/foo/justfile', came across a couple points that block this usage:
-
The main place I have such recipe is in a justfile that manages temporary clones of a couple repos. So the repo that contains the justfile I would use as module does not always exist. Currently this state causes
error: Failed to read justfilethat prevents running any recipes. Could there please be a way for "module only if specified file exists", i.e. if the module file is not found at the specified path, the justfile doesn't error out but the module/subcommands simply doesn't work? -
The documentation of this feature says -
justfile()andjustfile_directory()always return the path to the root
justfile and the directory that contains it, even when called from submodule
recipes.
This behavior means that justfiles with recipes that do just -f {{justfile()}} cannot be both usable standalone and usable as modules. Is there a way to get the full path of a justfile regardless of whether it's invoked directly or used as a module?
Hey @casey, thank you so much for just and for this update, quick question, do you plan, or maybe it's already possible and i've missed it to include modules from git repos, like @stereobutter initially mentioned, i would love that.
@WladyX Do you mean including modules from a local git repo, or a remote git repo?
Yes, exactly.
@WladyX Sorry, to be clear, do you want to load modules from a file in a local git repo, e.g., one that's on the same computer that just is running on, or a remote repo, like one on GitHub?
no worries Casey!
I would like to be able to import a module from a private gitlab instance.
My master plan is to a have repo with just modules and import the modules i want to any repo/folder i need.
eg:
- repo1 (aka just-modules) with modules foo1, foo2
- repo2 (aka myrepo) with bar that imports foo1 from repo1.mydomain.com
ideally i want to be able to make templates and by using variables to adapt them to each repo.
so when i update a template(module) i update it for every repo using it.
does that make sense?
I created #1799 to discuss remote imports.
I'm finding it to be off putting that the functions justfile() and justfile_directory() change behavior when in a modules file. I would expect that they always return the current file. Is this what module() and module_directory() are going to be used for?
Edit: I missed #929 (comment) already talking about this but just wanted to echo that is it causing me some grief as well.
Should an alias in a parent accept the names of recipes in modules? I am not sure if this might work in a future version of just, or whether it is not expected to be supported.
I am currently doing this in a parent justfile to enable top-level aliases to delegate to recipes in modules:
mod pre-commit
# Run all checks
lint:
@just pre-commit::checkIdeally, I would like to do something like this:
alias lint := pre-commit::check
Love the idea of this feature for better separation (and no naming clashes like import).
I found one potential issue (maybe I'm just using it wrong 😅), which I expected to work:
I've:
| .justfiles/fmt.just
| .justfiles/msrv.just
justfile
With content of .justfiles/clippy.just:
fmt: yare tests-integration tests-ui
yare:
cargo fmt --all
tests-integration:
cargo fmt --manifest-path {{justfile_directory()}}/yare-tests-integration/Cargo.toml
tests-ui:
cargo fmt --manifest-path {{justfile_directory()}}/yare-tests-ui/Cargo.toml
and .justfiles/msrv.just:
msrv: yare yare-macro
yare: install-cargo-msrv
cargo msrv verify --manifest-path {{justfile_directory()}}/Cargo.toml
yare-macro: install-cargo-msrv
cargo msrv verify --manifest-path {{justfile_directory()}}/yare-macro/Cargo.toml
install-cargo-msrv:
cargo install cargo-msrv --version 0.16.0-beta.20
and finally justfile:
mod fmt '.justfiles/fmt.just'
mod msrv '.justfiles/msrv.just'
before-push: fmt::fmt msrv::msrv
Potential issue A
I wanted to have before-push depend on the first command in '.justfiles/fmt.just' and the first command in '.justfiles/msrv.just', however, when either defining before-push: fmt::fmt msrv::msrv (path syntax) or before-push: fmt msrv (first command in module) I get:
just --unstable
error: Expected '&&', comment, end of file, end of line, identifier, or '(', but found ':'
——▶ justfile:6:17
│
6 │ before-push: fmt::fmt msrv::msrv
│ ^
Then, I figured, I would just call it instead of defining it as a dependency:
mod fmt '.justfiles/fmt.just'
mod msrv '.justfiles/msrv.just'
before-push:
fmt::fmt
Result:
just --unstable
fmt::fmt
sh: 1: fmt::fmt: not found
error: Recipe `before-push` failed on line 7 with exit code 127
Hmmm, maybe then:
mod fmt '.justfiles/fmt.just'
mod msrv '.justfiles/msrv.just'
before-push:
fmt
Result:
In this case, just doesn't return an error, but hangs indefinitely:
In the end I found this did work though:
mod fmt '.justfiles/fmt.just'
mod msrv '.justfiles/msrv.just'
before-push:
just --unstable fmt
just --unstable msrv
$ just --version
just 1.25.0
Recipes run external commands, so you likely have a binary called fmt in your PATH that waits for input on stdin, hence the hanging.
The way you've settled on (calling just <recipe>) is the only way I know of to call recipes across module boundaries.
I expect that eventually it will be possible to call module recipes as dependencies like you tried to.
Recipes run external commands, so you likely have a binary called
fmtin your PATH that waits for input on stdin, hence the hanging.
Ah of course! Too much tunnel vision I imagine, after I couldn't get the previous solutions to work.
The way you've settled on (calling
just <recipe>) is the only way I know of to call recipes across module boundaries.I expect that eventually it will be possible to call module recipes as dependencies like you tried to.
I think that would be incredibly useful. But even more so to be able to mark a recipe from a module as being dependent on another.
From #1202 (comment):
I want to give a final call for testing and feedback for the modules feature. All bugs I know of have been fixed
Would you consider #1866 to be a bug related to modules?
@crdx It's definitely a bug related to modules, although it can be fixed in a backwards-compatible way, so it isn't a stabilization blocker.
On my end, while impressed with the subcommand features implemented with modules and submodules, I am not really satisfied with the listing of the subcommands in modules and submodules.
I find it too verbose and too nested.
What I would expect, as it fits my taste, would be short lines for commands, and not showing the subcommands. So, I'd suggest, as an additional feature, a directive [no-unfold] or [fold-listing] (perhaps with a max nested level) that would only give the command giving access to subcommands, but not the nested subcommands themselves.
This way, I have the freedom to implement a help command in toplevel that would allow to invoke the listing of subcommands of a command and only that. Like just help my_command would get the listing of subcommands of my_command.
Working in small tmuxed fractions of screen over ssh, having a long, nested, list of commands, subcommands of commands, subcommands of subcommands of commands... not that good an idea.
I concur @gl-yziquel — here's a side-by-side comparison of the old behavior and one of the old-style way we have done since 2019 #208 (comment) and have since made way bigger:
And now I can't list the admin tasks with just --unstable admin -l or any such combination — I don't know if I'm doing something wrong or is it impossible with modules to first only show the top-level modules and then dive deeper like before.
The way we've made our just scripts discoverable and how we've taught that here is that we've told developers to first run just — then take a guess which "namespace" the thing you probably want belongs to, e.g. "start", and then run just start — and then pick the command which one thinks is correct. Here's an example of how one could figure out how to start a device simulator of any kind:
just- Oh, it might be "devices" — let's try it:
just devices - Ok cool, I'm interested in IE11 testing —
just devices ie11 - Ok now I know what to call
Again, I don't know if this behavior is reproducible with modules or not. But this is a crucial aspect for us to keep the scripts both maintainable and discoverable at the same time.''
The way we've made e.g. just devices work is that by convention, all of our Justfiles have their first script be help that only lists the available recipes in there:
I'd go as far as suggest the help behavior as in my example above should be the default and that showing the scripts inside modules should be an opt-in feature rather than having to supply something like [no-unfold] or [fold-listing] directives everywhere.
Also, spotted that using just --list --list-prefix "some prefix" with the modules prints out the "some prefix" list prefix double for modules. That looks like a bug — but I've never tested out list-prefix earlier. (I'm using just version 1.27.0 in all my screenshots here)
@BCNelson There are a bunch of new functions for getting paths. justfile() and justfile_directory() always return the path to the root justfile. source_file() and source_directory() return the path to the current file, regardless of whether it's a module or a import. module_file() and module_directory() will return the path to the current module. So inside an import, source_file() will return the path to the import file, and module_file() will return the path to the module that contains it. Note that module_file() and module_directory() are still unreleased.
@stuartellis and @foresterre Referring to parent and submodule recipes in aliases and dependencies is planned, but not for initial stabilization, since it can be added in a backwards-compatible way.
@gl-yziquel and @valscion Thanks so much for the feedback! I'm not a heavy user of modules (in fact, I don't think I've ever actually used one in my own justfiles 😅) so feedback is very helpful.
To sum up:
- the default display of submodule recipes is too verbose and nested
just --listshould accept a submodule path, so you can dojust --list SUBMODULEand it should behave as if you were doingjust --justfile SUBMODULE --list- it should be possible to iteratively descend into a tree of modules with
--list, so you can dojust --list, see stuff, then dojust --list SUBMODULEto see things in that submodule, etc
--list-prefix is working as intended. --list-prefix, by default, is four spaces, and you can use --list-prefix to change indentation.
--list output is human readable, not machine readable, so it isn't subject to backwards compatible guarantees, i.e., we can stabilize modules, but still change --list output in the future. That being said, I'd like to get it into a reasonable state before stabilizing modules.
I opened #2107 to track improving --list output with modules.
Something else I noticed, is that using the interactive chooser, e.g. @just --choose --unstable doesn't list anything from the modules (I expected it to).
I figured, if supplied with the module name like @just --choose --unstable dav1d it would show a list from the module, similar to how running just --unstable dav1d would execute the default script in the module dav1d.
@foresterre I wasn't actually able to reproduce this, but I opened #2110 to track it. If you could provide a repro there that would be great.
I just merged #2107, so --list no longer displays recipes in submodules, but instead displays MODULE ....
As far as I know, there are no current blockers to stabilization. There are a lot of features that I want to add later, but they can be added in a backwards compatible way, so they don't block stabilization.
My current plan is to stabilize modules on July 1st, so testing and feedback are much appreciated!