[FEATURE] Change backend configuration from Terramate CLI

[Crytlig]

Is your feature request related to a problem? Please describe.
My feature request is a direct frustration with Terraform's lack of variable usage in backend configuration blocks.
See hashicorp/terraform#13022 (all the way back from 2017 and still pending).

I have worked with Terraform for the past 5 years, and have followed Terramate on it's journey, and I like what I see!

What I currently struggle with is essentially very much summed up in the original issue mentioned before:
hashicorp/terraform#13022 (comment)

"Microservices are better versioned and managed discretely per component, rather than dumped into common prod/staging/dev categories which might be less applicable on a per-microservice basis, each one might have a different workflow with different numbers of staging phases leading to production release...."

I can see by the various examples given, for example by Sören in https://blog.terramate.io/how-to-keep-your-terraform-code-dry-by-using-terramate-be5807fef8f6, that the pattern that is strived for is to have my-project-prod/ my-project-stag/ etc.

This is fine and all for some applications, however, as my environments are on par with each other, I usually only need to change a few things in my variables, say var.env or var.location and change to a different backend to deploy to a new environment. That is usually handled by a wrapper script that is used in my automation engine of choice (GHA, Azure Pipelines, GitLab...).

Essentially, that breaks with the aforementioned pattern of duplicate files for dev, test, prod, you name it and allows me to deploy feature environments for me and my co-workers. Concretely, I deploy a new app service on Pull requests and deploy the code there until it's merged, and after merge it's deployed to dev, staging and production.

My wrapper script for feature deployments per pull request use a dynamically created key for backend configuration and is being set on -backend-config=key=<dynamic_key>, or if it's a smaller environment simply a path -backend-config=key=<path_to_developer_backend>.

I hope this describes the overall issue.

Using Terramate, I can no longer specify a specific path or a custom dynamically created backend_config variable AFAIK.
To give an example, say I have two stacks entra (for Azure Entra) and resources (for resources scoped within an Azure subscription), running terramate run -- terraform init -backend-config=key="helloworld" would cause issues, as the two stacks now would refer to the same backend key. The same issue goes for terramate run -- terraform init -backend-config=key=infra/stacks/<path breaks here>.

Describe the solution you'd like
Another issue #951 sort of mentions the same problem. A solution, without having dug into your codebase, would be to simply to provide a terramate run -backend-config=backends/my_feature_backend -- terraform init. Since the engine is traversing stacks, I assume that it's already doing a -chdir=stacks/<arbitrary_stack> so the path would be from the stacks perspective.

Obviously, the feature backend files would have to be named the same across stacks, but I don't see why that should not be possible, especially if it can be generated from a top level. The same could be done with environment variables terramate run -backend-config=key=${env.stack}-$MYKEY -- terraform init.

Describe alternatives you've considered
Unfortunately, I still have to found a workaround for this. Of course, I could create N number of known feature environments and use Terramate's code generation for each known environment. As I described earlier, though, I am not particularly a fan of that approach. While it seems like this is the purpose, there is some affordance to Terramate and I would like to use it specifically for stacks to split up infrastructure and use the change detection system.

In addition, I could also leverage Terraform workspaces but that would break existing workflows quite a lot.

I am curious about hearing your opinions on this issue.