Add `Config` to the `resource.ReadRequest`

[Teemu Matilainen]

Module version

v1.4.1

Use-cases

Allow users to (optionally) specify a default value for a common argument in many resources and data sources.

I.e., the resources and data sources have optional and computed foo attribute. If not specified, the value from the provider attribute foo would be used. If even that is not specified, fail the run.

For example:

provider "my" {
  foo = "bar"
}

resource "my_example" "this" {
  # `foo` taken from the provider config
}

resource "my_example" "that" {
  # Explicitly set
  foo = "zap"
}

Attempted Solutions

I have thought and tried different solutions, but all have their issues. I might of course miss something here.

1. Defaults are applied before the resource is configured, so it doesn't have access to the provider configuration.
2. Attribute Plan Modifier doesn't have access to the resource and thus provider configuration.
3. Resource Plan Modifier has all the needed configuration, but if foo is configured in the provider and changes, updating the resource will lead to the Provider produced inconsistent result after apply error on other computed fields. In a single resource it would be fine to set all to unknown in the ModifyPlan(), but no idea how to iterate them in a generic way if the same function is used in many resources.
4. Setting the default in Create() works. But in Read() there is no access to the Config and the State has the old value, so if the value comes from the provider, any changes are not detected. Another downside is that in diffs the value is always (known after apply).
5. Using another attribute for the actual value sounds verbose, and not sure it would help in many of the previous cases.
6. With data sources Read() seems like the only option, but there it works fine.

Proposal

Add Config to the resource.ReadRequest struct to be accessible in the resource's Read() function.

Of course if there are other ways to implement this, nice. 🙂
Some way for setting all the computed, null configured attributes back to unknown at least would be fine.

[Brian Flad]

Hi @tmatilai 👋 Thank you for raising this feature request.

The implementation reason that Config is not exposed in resource.ReadRequest is because this information is not exposed in the protocol between Terraform and providers:

https://github.com/hashicorp/terraform/blob/3420fdd5167c40c04aa79a8170885d5a0148f6ac/docs/plugin-protocol/tfplugin6.4.proto#L308-L321

The comments in the protocol definition mention some of the design reasoning for this:

This message intentionally does not include configuration data as any configuration-based or configuration-conditional changes should occur during the PlanResourceChange RPC. Additionally, the configuration is not guaranteed to be wholly known nor match the given prior state, which could lead to unexpected provider behaviors for practitioners.

Since that is a design problem that would need to be accepted as a design tradeoff (or somehow solved) upstream in Terraform, then implemented in the protocol, there is nothing that can be implemented in terraform-plugin-go or this Go module for the given proposal at the moment.

For the situation you mention though, I believe we may need some additional information before we can provide guidance. Is the optional and computed attribute returned from the API? If not, then this is not a situation that managed resource refresh (resource Read in framework terms) in Terraform was designed for -- it is designed to refresh Terraform's resource state from the "real world" resource state, generally from the remote system.

[Teemu Matilainen]

Hi Brian, thanks a lot for the detailed response! Even though I've been using Terraform for a very long time, unfortunately I've never studied the details of the internals. But learning bit by bit now. 🙂

I believe we may need some additional information before we can provide guidance.
Is the optional and computed attribute returned from the API?

Yeah sorry, I should have included more concrete example. I've seen this need already in a couple of places, but now it came with the TFE provider. Most of the resources include organization argument, which can also be configured in the provider. So the value is not coming from the API, but it's a required attribute for many API calls.

Almost all of the resources in the TFE provider still use the SDKv2, and now I came to this when trying to implement a new resource and data source using the plugin framework. And turns out that the existing implementation also suffers from the same issue.

Indeed Read was not my first place trying to implement this. But as said, all the other options had their challenges, too. 😅

Any suggestions highly appreciated. But I understand if the protocol specification doesn't allow perfect solution for this.

[Emond Papegaaij]

We also face an issue were we need access to the Config from the ReadRequest. Our API dynamically adds some attributes only when asked for. So, for example, we have this (where foo is one of these dynamic attributes):

resource "my_example" "res" {
  id = 42
  foo = "bar"
}

The provider will create this resource and explicitly ask the API to include foo in the response. If it wouldn't do that, the API will return the resource with the foo attribute set to null, causing a mismatch in the expected stated. When the Read method is called on the provider, it looks in the state to see if foo is set. If it is, it again asks the API to include it in the response.

The problems start when you use import to import the resource with id 42 into the state. The import has no way of knowing which attributes to include, so when Read is called, it only fetches the naked resource. Now, when you create a plan, it will notice a difference because foo is unset. Doing a refresh doesn't help either, because the Read method is still called with only the state it currently has, which does not include foo. For both invocations of Read (after import or at refresh), we need the Config to be able to determine the attributes we need to fetch.

At the moment, the only way out of this situation is to manually edit the state and set the required attributes to some (incorrect) non-null value and do a refresh. I really see no way to implement this correctly in the provider other than requiring the user to explicitly list the required attributes at several locations, which would result in a very poor experience.