[FEATURE] First class support for modules (HCL code generation and orchestration)
cwe1ss opened this issue · comments
The current main focus of Terramate is on stacks. Once a directory contains a stack {}
-block, it can be used as a target for code generation, change detection and orchestration:
- You can use
generate_hcl
to generate any dynamic .tf file - e.g. to dynamically set backends, providers, or versions. - You can use the terramate CLI to execute other programs in any stack - e.g.
terramate run terraform init
.
This feature request proposes extending the Terramate functionality to regular modules.
In contrast to a stack, a module is a reusable set of Terraform resources, that is not meant to be planned/applied directly. Instead, it is referenced by one or many stacks with the appropriate variables for the target environment.
While modules are no target for "terraform plan/apply", there are still scenarios where modules would benefit from Terramate functionality:
Generate HCL
One might want to use the same variable definitions for a set of modules. For example, multiple modules might need a context.tf (from Cloud Posse) which allows passing common variables between modules and sub-modules.
With Terramate's "Generate HCL" feature, one could declare this file once and have it generated for each module, making the solution more DRY.
* modules
* module-a
* _generated_context.tf
* ...
* module-b
* _generated_context.tf
* ...
* context.tm.hcl
* stacks
* ...
# context.tm.hcl
generate_hcl "_generated_context.tf" {
content {
variable "global_prefix" {
type = string
}
}
}
The demonstrated _generated_context.tf
content is static, but there are also use-cases for generating dynamic HCL. Each module must declare its required_providers
with a source and minimum version. By using the generate_hcl
feature of Terramate, one could centralize the generation of these declarations to keep version numbers in one location and to keep the code DRY. This means, that modules could also benefit from using globals
with their lazy evaluation.
Generate File (with local file names)
It is already possible to generate regular files in arbitrary directories by setting context = root
- e.g.
generate_file "/file.txt" {
context = root
content = "something"
}
However, this only supports setting an "absolute" path starting from the repository root ("/some/folder/file.txt"), so it is not possible to place a tm.hcl-file inside a module and generate a file within the same module without specifying the directory (which is error-prone if the directory is renamed)
# This attempted workaround to create a .tf file within the module is NOT possible
generate_file "_generated_context.tf" {
context = root
content = <<-EOT
variable "my_variable" {
type = string
}
EOT
}
Terramate Run
There are multiple scenarios where it would be good to use "terramate run" to invoke another program in all modules - e.g.
terraform validate
to validate the modulesterraform-docs
to create documentation for all modules
Describe the solution you'd like
I don't know the internals of Terramate and I don't have a lot of experience with it yet, so I can only guess what a possible solution might be from a user's perspective:
- Introduce a first class
module {}
-block that inherits most of the features fromstack
(HCL code generation, globals, ...). - Introduce a
module
-context (analog to thestack
-context) - By having a separate
module
-block/concept it can be documented accordingly with its differences to stacks
Introducing this into the CLI is probably tricky. I think, the nicer integration would be to introduce "stack"/"module" sub-commands, but this would obviously be a breaking change.
With sub-commands, instead of terramate create
we could have terramate stack create
and terramate module create
and adjust the logic/documentation to its needs.
Other commands like fmt
, generate
(which already operate on all child directories) would not need stack/module sub-commands.
Another option would be to introduce e.g. a --module
option to integrate the logic into existing commands - e.g. terramate create --module modules/my-module
. However, this option might not work with other options (e.g. --after
, --before
) and it would make the documentation more complex, since it would have to describe the differences.
Describe alternatives you've considered
- Treating modules as stacks by including a
stack {}
block- HCL code generation is possible when you declare a
stack {}
block within your modules. To ensure modules aren't included when running anyterramate
statements, one can use tags to exclude modules. - However, this solution conflicts with the idea of "stacks" and their documentation. (they are not "runnable")
- HCL code generation is possible when you declare a
- Using
generate_file
withcontext = root
- It is possible to generate files in arbitrary directories with
generate_file
andcontext = root
. However,- this requires the file path to be absolute, so one can not generate a file in the "current directory"
- this doesn't support globals or any other HCL features (like dynamic blocks, ...)
- this doesn't support hierarchical code generation
- It is possible to generate files in arbitrary directories with
Additional context
Related Discord discussion
Hi @cwe1ss, as discussed in discord, this use case makes sense for repositories managing local modules that want to benefit from code generation.
Adding a possibility to orchestrate different commands by default or context seems reasonable too.
Thanks for creating the detailed issue here, please give us some time to investigate different ways of making the desired functionality available.
The proposed solutions make sense and will be considered and I will share a draft proposal asap.
The proposed solutions make sense and will be considered and I will share a draft proposal asap.
This definitely is not critical as there are some workarounds so please don't feel any pressure! I appreciate your work on this tool and I really have been liking it more and more over the last few weeks, so thank you very much! ❤️
It would be awesome to be able to do terramate run -C modules -- terraform test
:)
maybe also consider providing a metadata to be able to distinguish between modules and stacks when using generate_file
and generate_hcl
We are planning new feature to make this possible. It will not be possible as a first class feature but the combination of different features can lead to the desired outcome.
-
Terramate Scripts - a new experimental feature allowing to specify scripts in specific sub-trees e.g. a
terramate script run module test
could be defined in amodules
sub-tree and would execute commands only within this subtree even if called on top level. the same way in astacks/
subtree aterramate script run terraform apply
could be made available that will not be defined inmodules
and thus not include themodules/
subtree in execution. -
As mentioned before
tags
can be used to limit execution further when usingterramate run
. An upcoming feature will allow to create tags on any level and they will inherit through the hierarchy to stacks, removing the need to define the same tags for every stack. This will make it easy to tag all stacks with aterraform
tag and all modules with amodule
tag.
This combination will allow to either use terramate scripts
for predefined actions or run any command but limit execution to non-modules via terramate run --no-tags module
.
It would be awesome to be able to do
terramate run -C modules -- terraform test
this is possible as of today. With experimental terramate scripts you can even define a script inside the modules
directory to limit it's scope.
I personally made peace with the fact that my reusable modules also are "stacks" and the mentioned combination of scripts & tag inheritance will make it much easier to differentiate between these reusable modules and actual runnable stacks.
Having a separate module
-concept (basically as an alias to stack
) is probably not worth the additional complexity, so I personally consider this topic solved for my use-cases.
I leave it up to you to decide if this issue should stay open.
It would be awesome to be able to do
terramate run -C modules -- terraform test
this is possible as of today. With experimental terramate scripts you can even define a script inside the
modules
directory to limit it's scope.
If we put the scripts inside a modules directory, would we be able to detect the changes with --changed ? this is real need here.
If we put the scripts inside a modules directory, would we be able to detect the changes with --changed ? this is real need here.
Yes, --changed
will be fully supported. As a script only runs, where it is defined it would be an intersection of stacks that are changed and stacks the script is available for that would execute the code defined in the script.
Or in other words, on changes stacks a script is not defined for, the script will not run.