Consider Unit Testable Schema Validation
bflad opened this issue · comments
terraform-plugin-go version
v0.5.0
Use cases
Terraform Plugin SDK has long included the (*helper/schema.Provider).InternalValidate()
method, which:
... should be called in a unit test for any provider to verify before release that a provider is properly configured for use with this library.
Terraform CLI version 1.1.0 and later includes some additional schema validation upfront, instead of potentially erroring out later in workflows: hashicorp/terraform#28124
While providers implementing this library are generally encouraged to use the available acceptance testing framework (currently in Terraform Plugin SDK helper/resource), that type of testing is not guaranteed to be used, nor is it guaranteed to trigger some of the Terraform CLI-based validation.
Having a schema validation functionality within terraform-plugin-go should allow provider developers to receive feedback on schema issues more quickly and potentially without the extra burden of acceptance testing (which itself may require credentials, etc). Theoretically, downstream SDKs can also benefit by being able to call into this functionality.
Attempted solutions
Manually implementing schema validation unit tests.
Acceptance testing using Terraform Plugin SDK helper/resource.
Proposals
Schema Method
The tfprotov5.Schema
and tfprotov6.Schema
types can implement a new method, e.g.
func (s Schema) Validate(ctx context.Context) error { /* ... */ }
Which calls into further Validate
methods on SchemaBlock
and SchemaAttribute
.
terraform-plugin-go providers could then implement a unit test for each schema based on this method:
func TestExampleSchema(t *testing.T) {
t.Parallel()
err := exampleSchema.Validate(context.Background())
if err != nil {
t.Error("unable to validate schema: %s", err)
}
}
Providers could also introduce this check prior to returning GetProviderSchemaResponse
. Downstream SDKs could either implement wrapping unit testing or similar checks during GetProviderSchemaResponse
handling or tfprotov5/tfprotov6 type conversion
GetProviderSchemaResponse Method
Expanding the above, provide a method on top of the tfprotov5.GetProviderSchemaResponse
and tfprotov6.GetProviderSchemaResponse
types. e.g.
func (r GetProviderSchemaResponse) Validate(ctx context.Context) error {
err := r.Provider.Validate(ctx)
if err != nil {
return fmt.Errorf("unable to validate provider schema: %w", err)
}
for name, schema := range r.DataSourceSchemas {
err := schema.Validate(ctx)
if err != nil {
return fmt.Errorf("unable to validate data source %s schema: %w", name, err)
}
}
for name, schema := range r.ResourceSchemas {
err := schema.Validate(ctx)
if err != nil {
return fmt.Errorf("unable to validate resource %s schema: %w", name, err)
}
}
}
This could also be introduced later.
ProviderServer Method
Expanding the above, provide a method on top of the tfprotov5.ProviderServer
and tfprotov6.ProviderServer
types, e.g.
func (s ProviderServer) Validate(ctx context.Context) error {
resp, err := s.GetProviderSchema(ctx, &GetProviderSchemaRequest{})
if err != nil {
return fmt.Errorf("unable to get schema: %w", err)
}
err = resp.Validate(ctx)
if err != nil {
return fmt.Errorf("unable to validate schema: %w", err)
}
}
This, along with additional provider server validations, could also be introduced later. This may be the most intuitive place for provider developers wanting to implement unit testing similar to Terraform Plugin SDK, albeit it would be the most broad. Since the RPC request objects are not customizable in this proposal without modifying the function signature, it may not be future proof.
Additional Context
Terraform CLI 1.1.0 implements the following rules:
- Attribute
- Must be non-nil
- Must set Optional, Required or Computed
- Cannot set both Optional and Required
- Cannot set both Computed and Required
- Must set Type or NestedType
- Cannot set both Type and NestedType
- NestingSet NestedType may not contain attributes of cty.DynamicPseudoType
- Block
- Must be non-nil
- Must not have overlapping attribute and nested block names
- Must have valid name (
^[a-z0-9_]+$
) - MinItems and MaxItems must both be greater than zero
- MinItems and MaxItems must match in NestingSingle mode
- MinItems and MaxItems must be set to either 0 or 1 in NestingSingle mode
- MinItems and MaxItems cannot be used in NestingGroup mode
- MinItems must be less than or equal to MaxItems in NestingList/NestingSet mode
- NestingSet blocks may not contain attributes of cty.DynamicPseudoType
- MinItems and MaxItems must both be 0 in NestingMap mode
References
- https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema#Provider.InternalValidate
- hashicorp/terraform#28124
- https://github.com/hashicorp/terraform/blob/6530055d188d363bf64ff99b56bb00b350de2cfb/internal/configs/configschema/internal_validate.go
- hashicorp/terraform-plugin-framework#113