Cannot create CloudFunction with terraform app authorization

[Konstantin Burtsev]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.18
provider.google v2.20.1

Affected Resource(s)

• google_cloudfunctions_function

Terraform Configuration Files

resource "google_cloudfunctions_function" "cloud-sql-exporter" {
  name    = "cloud-sql-exporter"
  runtime = "python37"

  available_memory_mb   = 256
  source_archive_bucket = google_storage_bucket.functions-code.name
  source_archive_object = google_storage_bucket_object.code-archive.name
  trigger_http          = true
  entry_point           = "main"

  environment_variables = {
    DB_INSTANCE   = "${module.module_name.db_instance_name[0]}",
    BACKUP_BUCKET = "${google_storage_bucket.project-db-backup.name}"
  }
}

Debug Output

Panic Output

Expected Behavior

gcloud function successfully created

Actual Behavior

google cloud function failed to create 

google_cloudfunctions_function.cloud-sql-exporter: Creating...

Error: Error waiting for Creating CloudFunctions Function: error while retrieving operation: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the cloudfunctions.googleapis.com. We recommend that most server applications use service accounts instead. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/., accessNotConfigured

  on functions.tf line 28, in resource "google_cloudfunctions_function" "cloud-sql-exporter":
  28: resource "google_cloudfunctions_function" "cloud-sql-exporter" {

Steps to Reproduce

1. terraform apply

Important Factoids

terraform authenticated as application with gcloud auth application-default login
Actually I don't have any credentials config within terraform.
BTW, I have successfully created cloud function couple of weeks ago (before NY2020) with same config, but now I see error. In fact - cloud functions created, but I cannot create dependent resources. I tried different provider versions (3.0.0-beta.1 and 3.4.0) with the same result.

References

• #0000

[courteouselk]

I have a similar situation. Only I could circumvent the problem by using service account.

[Calvin Behling]

My workaround is to actually call gcloud as a local-exec provisioner. Super janky but it works (only for creation).

resource "null_resource" "status_update" {
  triggers = {
    hash = data.archive_file.functions.output_base64sha256
  }

  provisioner "local-exec" {
    interpreter = ["bash", "-c"]
    command = join(" ", [
      "gcloud functions deploy",
      "${local.function_name}",
      "--project=${var.project}",
      "--region=${var.region}",
      "--runtime=go111",
      "--entry-point=EntryPoint",
      "--trigger-topic=${local.status_update_topic}",
      "--source=gs://${local.function_bucket_name}/${google_storage_bucket_object.functions.name}",
      "--set-env-vars=${local.function_env_vars}",
      "--max-instances=1",
      "--timeout=60",
      "--memory=128MB",
      "<<< 'N\n'",
    ])
  }
}

[Edward Sun]

@poligraph it seems to me this is caused by the account you used and its permission. Can you go to https://www.terraform.io/docs/providers/google/guides/getting_started.html and follow the steps under Adding credentials? Make sure you give the new service account with Cloud Functions Admin role. Please let me know if you still have the problem? Thanks

You mentioned Actually I don't have any credentials config within terraform.. Is there particular reason you did it this way?

[Konstantin Burtsev]

@edwardmedia thanks for response, I use service account called "Google APIs Service Agent" with name [e-mail] and authenticate with gcloud auth application-default login and actually there is no json key configuration for terraform. I have quite big config and all modules work fine except cloud_functions.

[Edward Sun]

@poligraph can you verify if the account you are using has that role? We would recommend using a separate account. The preferred method of provisioning resources with Terraform is to use a GCP service account, a "robot account" that can be granted a limited set of IAM permissions. You don't want to mess up with the credential for gcloud

[Konstantin Burtsev]

@edwardmedia I tried to use GOOGLE_CLOUD_KEYFILE_JSON with service account "Editor" permissions and it worked. But anyway, is it any possibility to look through the original issue, that cloudfunctions.googleapis.com could not be created via Google APIs Service Agent. I manage tens projects in gcloud and it is not convenient to use service account json for every single project. Thanks.

[Edward Sun]

@poligraph your question is beyond what terraform provider can do. The account you chosen might not be best fit for running terraform. Please take a look below comments about this account. Besides, You should be able to create a specific service account that have permissions to all the projects you need to manage. It is not limited to just one project.

"Google APIs service account
An example of a Google-managed service account is a Google API service account identifiable using the email:

[e-mail]

This service account is designed specifically to run internal Google processes on your behalf and is not listed in the Service Accounts section of the Cloud Console. By default, the account is automatically granted the project editor role on the project and is listed in the IAM section of the Cloud Console. This service account is deleted only when the project is deleted. Google services rely on the account having access to your project, so you should not remove or change the service account's role on your project."

[Casper de Groot]

I'm not certain, but is this happening when you use google-project-factory v6.2.0 or v6.2.1?
For me, it worked again with v6.1.0.
(I'm working on a simplified testcase so i can file a bug, but I'm not certain yet)

[Federico Delgado Doyle]

I've used the exact same account for the last 5 months (at least). But this issue showed up like a week ago or a bit more.

Error: Error waiting for Deleting CloudFunctions Function: error while retrieving operation: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the cloudfunctions.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/., accessNotConfigured

while a service account is clearly suggested. I still don't get why the error shows up now. Is it a change gcp side? I don't remember updating any provider.

[Edward Sun]

@flaker if you take a closer look at the error message, it appears your calls were rejected by googleapi. Sorry, we don't have much I can provide related to the API changes.

error while retrieving operation: googleapi: Error 403

[Joshua Baxter]

What's strange about this issue is that the function gets created fine. If you run the deploy a second time it will complete successfully. Using a service account works, but i dislike having service account keys kept around on devs machines. You also lose the audit trail of who has done what. Impersonation works fine too, but again, no audit trail in the logs as to who has actually modified or created infrastructure.

[Edward Sun]

@schaffino Need audit trail is legit. Let's see what we can do.

[Dana Hoffman]

As @edwardmedia noted, we don't recommend using gcloud credentials to authenticate to Terraform, because it opens the doors for this exact style of breakages. From https://www.terraform.io/docs/providers/google/guides/provider_reference.html#credentials-1:

On your computer, you can make your Google identity available by running gcloud auth application-default login. This approach isn't recommended- some APIs are not compatible with credentials obtained through gcloud.

If you can't use a service account, you should be able to succeed by using an access token, with the important caveat that it will expire. I also can't guarantee that it won't at some point end up with a similar error as when using application credentials.

If neither of these are options, then you should obtain end-user credentials using an application that is not gcloud (since its credentials are really meant to be used in gcloud itself). I haven't tried this myself, but I think it would require setting up an oauth application that you can then get a credentials file from. https://cloud.google.com/docs/authentication/end-user would be the place to look to get started. If you try it, let me know how it goes!

[anativ]

I have the same issue
it works when the provider = google but fail when provider = google-beta
It creates the cloud function but I get an error:

Error waiting for Creating CloudFunctions Function: error while retrieving operation: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the cloudfunctions.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/., accessNotConfigured

Same project / same service account

[Federico Delgado Doyle]

Hi @edwardmedia . Yes, I noticed the 403. Checking the debug logs, I see the function being created (and I can also see that in the admin console) but the subsequent call to check the status of the creation failing (403). That seems counter intuitive right? the function creation was allowed (output seems to indicate everything failed). Plus I used this for several months working before it stopped suddenly. At first I thought it was some incompatibility with the provider, but it didn't matter which version I tried. There was a change somewhere else.

[Dana Hoffman]

@flaker indeed this was an API-side change rather than a Terraform one. The comment above yours and the one above that mention some workarounds, so hopefully one of those will work for you. I agree that the behavior is counterintuitive.

For anyone on this thread that's a Googler or has a Google-side contact, you (/they) can follow b/148479480 for updates.

[Deleted user]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [e-mail]. Thanks!