Invoke-Build "subtasks"?

[dsbenghe]

Hi,

We are using Invoke-Build for our .net/C# projects. We started to use it for even more than just builds but as a general cli entry point for automating the developer workflow.

Due to the many tasks and associated parameters we thought to split/structure it in one root script and multiple "sub-root" scripts based on the different functionalities provided with tasks and subtasks i.e.

build.ps1 with tasks build, deploy, infrastructure
src/build.ps1
deploy/build.ps1
infrastructure/build.ps1

and got to root build.ps1 having tasks build, deploy, infrastructure which are dispatching to subtasks (the inner builds tasks)

build.ps1 ? = build, deploy, infrastructure
build.ps1 build ? = compile, test, package, ...
build.ps1 deploy ? = create-db,

e.g. build.ps1 build test -Configuration:Debug

• task test and parameter Configuration defined in src/build.ps1
• task build defined in build.ps1 and dispatching to src/build.ps1

That seems to work reasonable well, except:

• parameters suggestions for inner builds when used via root build
• argument completers when used via root build

Any PS/Invoke-Build magic which can help with this?

Or maybe some better way to structure a large number of tasks and params without pilling them up in the root script?

Thanks

[Roman Kuzmin]

Hi @dsbenghe It's good to know that IB works well in your hierarchy of sub-builds. I use somewhat similar systems, too, e.g. https://github.com/nightroman/Invoke-Build/wiki/Build-Scripts-in-Projects#farnet-family-c-c-powershell

To be on the same page, what do you mean by "when used via root build"? Any concrete example for the scenario and code line where you need "parameters suggestions" and "argument completers" to work?

Any PS/Invoke-Build magic which can help with this?

Maybe but do not be too optimistic. I implemented various straightforward helpers. As for other helpers... well, they are probably not straightforward :)

Or maybe some better way to structure a large number of tasks and params without pilling them up in the root script?

Again, I'd like to know the exact use case? Do you mean dot-sourcing or something else? Any detail may help.

[Roman Kuzmin]

Also you mention your scripts as build.ps1. The standard naming pattern is *.build.ps1. It's fine if you do not follow. But are they IB build scripts at all or some wrappers? Due to not standard naming you may miss argument completion provided, say, by https://www.powershellgallery.com/packages/Invoke-Build.ArgumentCompleters

[Roman Kuzmin]

I think it would be ideal if you attach a zip file with a few sample scripts with "subtasks".

[dsbenghe]

Thanks for your quick answer. I uploaded a sample, as asked, here - https://github.com/dsbenghe/InvokeBuildSubTasks

Basically hierarchical builds, how are used in FarNet & also our builds, are working well as the tasks are the same along the hierarchy and the params the same.

When trying to drive more than just builds with IB, the names of the tasks are different between the child scripts and params also. We can pack everything in the top scripts: params, tasks, but I'm asking myself if there is a better way to do this to capture that the child scripts are doing quite a different job with quite different task names and params.

This suggested the idea of tasks/subtasks (command/subcommands) i.e.

root.build.ps1 roottask
root.build.ps1 build task1 -BuildParam1 blah
root.build.ps1 build build-task2 -BuildParam2 blah
root.build.ps1 deploy task1 -DeployParam1 blah
root.build.ps1 deploy deploy-task2 -DeployParam2 blah

Which gives a nice way of structuring and driving the tasks/subtasks - in fact, the task build works mostly as a tasks group selector.

But to get that we need to copy all the params definitions from child builds to the root one. Of course, we can drop the tasks/subtasks idea and just get to call build.build.ps1 directly without the root script.

Maybe what I'm looking for is just not possible?

[Roman Kuzmin]

@dsbenghe Thank you for the excellent input. Let me digest and think if anything possible. I am interested in this development.

[Roman Kuzmin]

In theory it looks possible to support a conventional parameter, say $by:

param(
    [ValidateSet('deploy', 'src')]
    $by
)
...

When IB discovers script parameters and finds $by set to, say "deploy", then
it simply replaces the current script with the default in the "deploy" folder
and takes parameters and tasks from there.

So, without -by it is the usual root.build.ps1:

root.build.ps1 roottask

But with -by it totally delegates the call to the script in the "src" folder:

# for "src"

root.build.ps1 -by src task1 -BuildParam1 blah
root.build.ps1 -by src task2 -BuildParam2 blah

# for "deploy"

root.build.ps1 -by deploy task1 -DeployParam1 blah
root.build.ps1 -by deploy deploy-task2 -DeployParam2 blah

Also, with this approach we do not have to define these boilerplate dispatchers:

task build {
    # just delegates to build.build.ps1
    Invoke-Build -Task $SubTasks -File "src/build.build.ps1" @PSBoundParameters
}

task deploy {
    # just delegates to deploy.build.ps1
    Invoke-Build -Task $SubTasks -File "deploy/deploy.build.ps1" @PSBoundParameters
}

Completers for parameters and tasks should work accordingly (in theory, cannot be sure without trying).

What do you think? Is it what you are looking for, approximately?

[Roman Kuzmin]

Yet, a question: instead, and right now, with all features available, why not just call things like this:

root.build.ps1 roottask

src/build.build.ps1 task1 -BuildParam1 blah
src/build.build.ps1 build-task2 -BuildParam2 blah

deploy/deploy.build.ps1 task1 -DeployParam1 blah
deploy/deploy.build.ps1 deploy-task2 -DeployParam2 blah

[dsbenghe]

Yes, sure, we can use the individual scripts as mentioned above - but the more functionalities are provided via the scripts, the more scripts we have, the more confusing is getting to use them and the less self-discoverable are becoming. Having one entry point developing cli is a lot nicer. Anyone who has used a similar cli (kubectl for example) knows the niceness of using such a cli.

Yes. I think this is what I'm looking for - it will be better without -by, but I guess this is not possible without providing a "this" for the current script i.e. root.build.ps1 this roottask. But I must say I still don't see how command line PS will be able to list the parameters for src/build.build.ps1 for example to help discoverability (but here this could be due to my limited PS knowledge).

And I don't mind a little bit of boilerplate code if gets a better result.

[Roman Kuzmin]

Without an extra parameter -by (or different conventional name) I am not sure how to make this, technically. Suggestions?

But I must say I still don't see how command line PS will be able to list the parameters for src/build.build.ps1

In the same way as IB discovers parameters of build scripts and exposes them as "its own". In this case IB will be told to take parameters and tasks not from the current script but from the -by script.

And I don't mind a little bit of boilerplate code if gets a better result.

Well... in the described scenario the boilerplate is simply not possible. If you use -by then the whole current script is not used, it just works as a dispatcher to the script specified by -by.

[Roman Kuzmin]

Please note that I am just exploring the area, I have not yet decided anything. I do not know if I like it in this form.
In fact, I am not even sure that the suggested way will work (I started to have doubts...)

[Roman Kuzmin]

See my fork https://github.com/nightroman/InvokeBuildSubTasks
Good news: some tweaks are possible to make sub task and sub parameters available for code completion.

So it is possible to compose this command with helpers (task1 is tab-completed), (-DeployParam1 is tab-completed):

.\root.build.ps1 deploy task1 -DeployParam1 zzz

Bad news: it is not possible to invoke such a command:

A parameter cannot be found that matches parameter name 'DeployParam1'.

because the actual invocation is done by Invoke-Build and it does not see these dynamic parameters.

[Roman Kuzmin]

OK, this workaround takes us one step further to the next issue. We now can invoke

.\root.build.ps1 deploy task1 -DeployParam1 zzz

but it does not receive the parameter zzz:

Build deploy C:\-\InvokeBuildSubTasks\root.build.ps1
Task /deploy
Build task1 C:\-\InvokeBuildSubTasks\deploy\deploy.build.ps1
Task /task1
child 2 task 1                                 <----- expected "child 2 task 1 zzz"
Done /task1 00:00:00.0060011
Build succeeded. 1 tasks, 0 errors, 0 warnings 00:00:00.0480013
Done /deploy 00:00:00.0689999
Build succeeded. 2 tasks, 0 errors, 0 warnings 00:00:00.1789995

[Roman Kuzmin]

OK, and with this change it all kind of works :)

And I don't mind a little bit of boilerplate code if gets a better result.

It's not a little bit but it looks doable after all.
What do you think?

[dsbenghe]

Looks that it works exactly how I thought about. Great work.

[Roman Kuzmin]

I am glad to hear that! Closing?

P.S. I will probably create a use case in https://github.com/nightroman/Invoke-Build/tree/master/Tasks and cover by tests.

[Roman Kuzmin]

@dsbenghe I've added the "show case" -- https://github.com/nightroman/Invoke-Build/tree/master/Tasks/SubTasks

Any remarks, other ideas? If it's done I'll close the issue and remove my fork repository. Before that, I may submit a PR to your original repo if you like. Please let me know.

Thank you for the interesting case.

[dsbenghe]

You can close it. Feel free to create a PR to the original repo. If not I will merge it myself. Thanks.

[Roman Kuzmin]

I've sent the PR. Closing this.