SIGSEGV when using GKE Workload Identity

[Dominik Siebel]

Our terraform state started segfaulting on every plan after the recent upgrade to the newest terraform version and the newest version of the gsuite provider.

Background:
We are using the GSuite provider in a rather large terraform state.
plan and apply are executed using Atlantis running on Google Kubernetes Engine authenticating via Workload Identity.
I recently updated the entire stack

• Terraform 0.13.6 to 0.15.1
• Atlantis 0.16 to 0.17

Since the upgrade all plans fail with a SIGSEGV in the GSuite provider.
I encountered multiple segmentation violations in the past using this provider and was always able to work around them, this time I have no clue where to start.

The gsuite provider was used with ~> 0.1.54 version constraint so I assume that we were already using 0.1.60 shortly after it was released.

Error output:

Error: Plugin did not respond

  with provider["registry.terraform.io/deviavir/gsuite"],
  on main.tf line 45, in provider "gsuite":
  45: provider "gsuite" {

The plugin encountered an error, and failed to respond to the
plugin.(*GRPCProvider).ConfigureProvider call. The plugin logs may contain
more details.

Stack trace from the terraform-provider-gsuite_v0.1.60 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xd8170f]

goroutine 65 [running]:
github.com/DeviaVir/terraform-provider-gsuite/gsuite.(*Config).loadAndValidate(0xc0003305b0, 0xc000796038, 0x6, 0xe47280, 0x1662620)
	/go/src/github.com/DeviaVir/terraform-provider-gsuite/gsuite/config.go:125 +0x62f
github.com/DeviaVir/terraform-provider-gsuite/gsuite.providerConfigure(0xc0003303f0, 0xc000796038, 0x6, 0xc0003303f0, 0x0, 0x0, 0x0)
	/go/src/github.com/DeviaVir/terraform-provider-gsuite/gsuite/provider.go:134 +0x37e
github.com/DeviaVir/terraform-provider-gsuite/gsuite.Provider.func1(0xc0003303f0, 0x0, 0xc000564360, 0xc0003303f0, 0x0)
	/go/src/github.com/DeviaVir/terraform-provider-gsuite/gsuite/provider.go:76 +0x58
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).Configure(0xc0005a2100, 0xc0001e2660, 0xea1720, 0xc0001e25a0)
	/go/src/github.com/DeviaVir/terraform-provider-gsuite/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/provider.go:275 +0xfc
github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).Configure(0xc0005a00a0, 0x1162bb0, 0xc000324a80, 0xc0005a6480, 0xc0005a00a0, 0xc000324a80, 0xc0000cfba0)
	/go/src/github.com/DeviaVir/terraform-provider-gsuite/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:487 +0x2ee
github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_Configure_Handler(0xf9a4e0, 0xc0005a00a0, 0x1162bb0, 0xc000324a80, 0xc0000bc300, 0x0, 0x1162bb0, 0xc000324a80, 0xc00022a140, 0x12c)
	/go/src/github.com/DeviaVir/terraform-provider-gsuite/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3251 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003b7340, 0x116cff8, 0xc000583800, 0xc0004f8100, 0xc000116990, 0x16c3c78, 0x0, 0x0, 0x0)
	/go/src/github.com/DeviaVir/terraform-provider-gsuite/vendor/google.golang.org/grpc/server.go:1217 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0003b7340, 0x116cff8, 0xc000583800, 0xc0004f8100, 0x0)
	/go/src/github.com/DeviaVir/terraform-provider-gsuite/vendor/google.golang.org/grpc/server.go:1540 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0003d0290, 0xc0003b7340, 0x116cff8, 0xc000583800, 0xc0004f8100)
	/go/src/github.com/DeviaVir/terraform-provider-gsuite/vendor/google.golang.org/grpc/server.go:878 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/DeviaVir/terraform-provider-gsuite/vendor/google.golang.org/grpc/server.go:876 +0x1fd

Error: The terraform-provider-gsuite_v0.1.60 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[Dominik Siebel]

UPDATE: The segfault occurs after some sort of timeout, trying to authenticate.
I switched our setup from using Workload Identity back to using the JSON key of the Service Account directly.

Still, there seems to be an issue in the provider when a timeout occurs in the auth process, causing the SIGSEGV.
I adjusted the title accordingly.

[Dominik Siebel]

Did some digging through the code, it looks like the combination empty Credentials (should be empty because Workload Identity doesn't require us to set GOOGLE_APPLICATION_CREDENTIALS or similar)
https://github.com/DeviaVir/terraform-provider-gsuite/blob/v0.1.60/gsuite/config.go#L62
and ImpersonatedUserEmail being set
https://github.com/DeviaVir/terraform-provider-gsuite/blob/v0.1.60/gsuite/config.go#L96
leaves client without any value, which then is accessed to assign client.Transport
https://github.com/DeviaVir/terraform-provider-gsuite/blob/v0.1.60/gsuite/config.go#L125

[Dominik Siebel]

Looks like this issue was fixed in #175
and released as https://github.com/DeviaVir/terraform-provider-gsuite/releases/tag/v0.1.61

Sadly v0.1.61 is not yet available via the public Terraform registry. Any ETA when this will be available?

[Chase]

Strange, it's complaining about missing release asset for ["terraform-provider-gsuite_0.1.61_SHA256SUMS"] which is available in the tagged release

[Dominik Siebel]

Any chance you can fix it? 😜

[Chase]

Apparently pressing Resync a few times a day fixes it. https://registry.terraform.io/providers/DeviaVir/gsuite/latest

[Dominik Siebel]

Thank you very much, will try it asap!