[BUG] Script commands are executed in wrong order when using complex `after`-dependencies

[Christian Weiss]

Describe the bug
Script commands are executed in wrong order for some stacks when I'm using stacks in different parent folders and when they depend on each other by using after-statements.

To Reproduce

Here's the repo to reproduce the issue: https://github.com/cwe1ss/terramate-scripts-bug

The stacks in this repo have a lot of after-definitions:

• /management/stack-a: after "../stack-b"
• /management/stack-b: (no after statement)
• /management/stack-c: after "../stack-a"
• /network/stack-a: after "../../operations/stack-a"
• /network/stack-b: (no after statement)
• /operations/stack-a: after "../../management/stack-c"
• /operations/stack-b: (no after statement)

This results in the following terramate list --run-order output:

stacks/management/stack-b
stacks/management/stack-a
stacks/management/stack-c
stacks/operations/stack-a
stacks/network/stack-a
stacks/network/stack-b
stacks/operations/stack-b

The output for terramate script run drift is as follows. As you can see, sometimes "cmd-2" is printed before "cmd-1".

Script 1 at /imports/scripts.tm.hcl:1,1-9,2 having 1 job(s)
/stacks/management/stack-b (script:1 job:0.0)> cmd.exe /C echo cmd-1
cmd-1
/stacks/management/stack-b (script:1 job:0.1)> cmd.exe /C echo cmd-2
cmd-2
/stacks/management/stack-a (script:1 job:0.1)> cmd.exe /C echo cmd-2
cmd-2
/stacks/management/stack-a (script:1 job:0.0)> cmd.exe /C echo cmd-1
cmd-1
/stacks/management/stack-c (script:1 job:0.0)> cmd.exe /C echo cmd-1
cmd-1
/stacks/management/stack-c (script:1 job:0.1)> cmd.exe /C echo cmd-2
cmd-2
/stacks/operations/stack-a (script:1 job:0.0)> cmd.exe /C echo cmd-1
cmd-1
/stacks/operations/stack-a (script:1 job:0.1)> cmd.exe /C echo cmd-2
cmd-2
/stacks/network/stack-a (script:1 job:0.0)> cmd.exe /C echo cmd-1
cmd-1
/stacks/network/stack-a (script:1 job:0.1)> cmd.exe /C echo cmd-2
cmd-2
/stacks/network/stack-b (script:1 job:0.1)> cmd.exe /C echo cmd-2
cmd-2
/stacks/network/stack-b (script:1 job:0.0)> cmd.exe /C echo cmd-1
cmd-1
/stacks/operations/stack-b (script:1 job:0.0)> cmd.exe /C echo cmd-1
cmd-1
/stacks/operations/stack-b (script:1 job:0.1)> cmd.exe /C echo cmd-2
cmd-2

Expected behavior
cmd-1 should always be executed before cmd-2.

Environment (please complete the following information):

• OS: Windows (for the repo, but also happened via Ubuntu dev container)
• Git Version: 2.32.0
• Terramate Version: 0.4.5

Additional context
https://discord.com/channels/1088753599951151154/1208039550559981578

[Sebastian]

Thanks for the detailed report! 👍

I already fixed it recently in main with #1444 after realizing it could happen theoretically, but I was not able to reproduce it. With your example I can, and I can also confirm that the fix works:

(v0.4.5)

go run github.com/terramate-io/terramate/cmd/terramate@v0.4.5 script run drift
Script 1 at /imports/scripts.tm.hcl:1,1-9,2 having 1 job(s)
/stacks/management/stack-b (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/management/stack-b (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/management/stack-a (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/management/stack-a (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/management/stack-c (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/management/stack-c (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/operations/stack-a (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/operations/stack-a (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/network/stack-a (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/network/stack-a (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/network/stack-b (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/network/stack-b (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/operations/stack-b (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/operations/stack-b (script:1 job:0.1)> echo cmd-2
cmd-2

(latest main)

go run github.com/terramate-io/terramate/cmd/terramate@main script run drift 
Script 1 at /imports/scripts.tm.hcl:1,1-9,2 having 1 job(s)
/stacks/management/stack-b (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/management/stack-b (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/management/stack-a (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/management/stack-a (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/management/stack-c (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/management/stack-c (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/operations/stack-a (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/operations/stack-a (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/network/stack-a (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/network/stack-a (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/network/stack-b (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/network/stack-b (script:1 job:0.1)> echo cmd-2
cmd-2
/stacks/operations/stack-b (script:1 job:0.0)> echo cmd-1
cmd-1
/stacks/operations/stack-b (script:1 job:0.1)> echo cmd-2
cmd-2

[i4k]

@snakster Amazing.
Then I think it would be nice to use @cwe1ss example as a testcase for avoiding a regression in the future.