Protocol Data Files Overwritten

Question

Protocol Data Files Overwritten

bflad opened this issue 2 years ago · comments

terraform-plugin-go version

v0.14.2

Expected Behavior

When enabling protocol data file logging via TF_LOG_SDK_PROTO_DATA_DIR, file names are not overlapping and contents are not overwritten.

Actual Behavior

Since the file names have second-level granularity, they can overlap if multiple Terraform requests for the same RPC occur within the same second, such as running provider acceptance testing of a state-only provider. In the below log capture, the first PlanResourceChange is a test step that fails, while the second PlanResourceChange is the implicit destroy test step. I was trying to view the data from the failing test step, but all that is on disk is the overwritten data.

2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Received request: tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Writing protocol data file: tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_Config.msgpack tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85 tf_provider_addr=registry.terraform.io/hashicorp/hashicups
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Wrote protocol data file: tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_Config.msgpack tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Writing protocol data file: tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_PriorState.msgpack tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Wrote protocol data file: tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_PriorState.msgpack tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Writing protocol data file: tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_ProposedNewState.msgpack tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85 tf_provider_addr=registry.terraform.io/hashicorp/hashicups
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Wrote protocol data file: tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_ProposedNewState.msgpack tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Writing protocol data file: tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_ProviderMeta.empty tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Wrote protocol data file: tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_ProviderMeta.empty tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85 tf_provider_addr=registry.terraform.io/hashicorp/hashicups
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Writing protocol data file: tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_PriorPrivate.empty tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85 tf_provider_addr=registry.terraform.io/hashicorp/hashicups
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Wrote protocol data file: tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_PriorPrivate.empty tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order
2023-01-04T09:07:43.257-0500 [TRACE] sdk.proto: Sending request downstream: tf_req_id=afa11208-95e1-d285-1b9f-f2ad92b61d85 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_PriorPrivate.empty
...
2023-01-04T09:07:43.313-0500 [TRACE] sdk.proto: Received request: tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order
2023-01-04T09:07:43.313-0500 [TRACE] sdk.proto: Writing protocol data file: tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_Config.msgpack tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups
2023-01-04T09:07:43.313-0500 [TRACE] sdk.proto: Wrote protocol data file: tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_Config.msgpack
2023-01-04T09:07:43.313-0500 [TRACE] sdk.proto: Writing protocol data file: tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_PriorState.msgpack tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups
2023-01-04T09:07:43.314-0500 [TRACE] sdk.proto: Wrote protocol data file: tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_PriorState.msgpack tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3
2023-01-04T09:07:43.314-0500 [TRACE] sdk.proto: Writing protocol data file: tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_ProposedNewState.msgpack
2023-01-04T09:07:43.314-0500 [TRACE] sdk.proto: Wrote protocol data file: tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_ProposedNewState.msgpack tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3
2023-01-04T09:07:43.314-0500 [TRACE] sdk.proto: Writing protocol data file: tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_ProviderMeta.empty tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order
2023-01-04T09:07:43.314-0500 [TRACE] sdk.proto: Wrote protocol data file: tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_ProviderMeta.empty tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3
2023-01-04T09:07:43.314-0500 [TRACE] sdk.proto: Writing protocol data file: tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_PriorPrivate.empty tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order
2023-01-04T09:07:43.314-0500 [TRACE] sdk.proto: Wrote protocol data file: tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_PriorPrivate.empty tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups
2023-01-04T09:07:43.314-0500 [TRACE] sdk.proto: Sending request downstream: tf_proto_data_file=/tmp/framework603/1672841263_PlanResourceChange_Request_PriorPrivate.empty tf_req_id=554d9c11-105b-71a8-8f89-f68580b60300 tf_provider_addr=registry.terraform.io/hashicorp/hashicups tf_proto_version=6.3 tf_rpc=PlanResourceChange tf_resource_type=hashicups_order

Steps to Reproduce

In a state-only provider:

TF_LOG_SDK_PROTO_DATA_DIR=/tmp go test -count=1 -v ./...

Brian Flad · Answer 1 · Wed Jan 04 2023 22:41:22 GMT+0800 (China Standard Time)

A quick workaround for this is adding a time.Sleep(1*time.Second) in the provider code (e.g. resource Read method).

A quick proposal fix for this is switching from time.Now().Unix() to time.Now().UnixMilli() for the time portion of the file name. A potentially more ideal proposal fix for this is appending the SDK generated request ID (tf_req_id) to the file names before the file extension, however it may not be straightforward to extract that from the terraform-plugin-log managed part of the context.Context.