Latest version of resource not found for state created outside of Concourse

[Mick Sear]

Hi there. Thanks for your work on this useful image.

I have a Concourse GCP project that I create using CloudBuild and Terraform. I want to use an output of that terraform state in a concourse pipeline, so I defined a resource like this:

- name: build-tf
  type: terraform
  source:
    env:
      GOOGLE_CREDENTIALS: "((gcp.concourse_sa))"
    backend_type: gcs
    backend_config:
      bucket: xyz-terraform
      prefix: terraform/state/build
      credentials: ((gcp.concourse_sa))
    vars:
      postgres_user_password: ((gcp.cloudsql_pw))

Later, I have a plan with this:

  - in_parallel:
      - get: git-platform-infra
      - get: build-tf
      - get: dev-tf
      - get: qa-tf
      - get: prod-tf

If I start a build manually, all the other resource versions are correctly detected apart from the one created outside of Concourse: The build-tf.

If I run fly check-resource, it succeeds:

$ fly -t infra check-resource --resource shared-env/build-tf
id      name       status     check_error
212867  terraform  succeeded

However, in the concourse UI, trying to run the pipeline, I get:

waiting for a suitable set of input versions
>  build-tf - latest version of resource not found

Is this a Concourse issue or an issue with terraform-resource?

Thanks

[Mick Sear]

I found that this works by setting env_name: default.

[Lyle Franklin]

@micksear glad you figured it out! Yeah, checking for out-of-band statefile changes is currently a limitation in the resource:

terraform-resource/src/terraform-resource/check/check.go

Lines 27 to 29 in 681e43c

 // Triggering on new versions is only supported in single-env mode: 
 // - expensive to check for changes across all statefiles 
 // - triggering on changes to any environment doesn't seem very useful 

. I'd consider accepting a PR that fixes that limitation if there was a strong use-case, but so far I've only seen people need to do this as a one-off migration, e.g. creating a new CI pipeline with existing statefiles. In these cases, check-resource + env_name as you've described will workaround the issue.