blob/gcsblob: anonymous bucket access still requires default credentials

[Caleb Brown]

Describe the bug

The GCS blob implementation requires default credentials to exist, even if the only buckets being used are anonymous (i.e. access_id=-).

This is due to lazyCredsOpener.OpenBucketURL (see code) calling gcp.DefaultCredentials(ctx) during its initialization.

To Reproduce

1. Ensure the env variable GOOGLE_APPLICATION_CREDENTIALS is not set.
2. Ensure the current user does not have ~/.config/gcloud/application_default_credentials.json set.
3. Ensure you are not running inside GCP (with a metadata service).
4. Attempt to call _, err := blob.OpenBucket(context.Background, "gs://some-bucket-example?access_id=-").
5. err will not be nil. When printed it shows:

open bucket gs://some-bucket-example?access_id=-: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information

Expected behavior

GCP credentials are not attempted to be read, unless a GCS url without access_id=- is passed to OpenBucketURL.

So, if all GCS urls have access=-, gcp.DefaultCredentials(...) is never called.

Version

v0.33.0

Additional context

n/a

[Caleb Brown]

A workaround for this is to create a dummy ADC file that satisfies gcp.DefaultCredentials(...), and use that instead.

For example:

echo '{ "type": "service_account" }' > dummy_adc.json
export GOOGLE_APPLICATION_CREDENTIALS=dummy_adc.json

[Robert van Gent]

Can you give it a try after #3306?

[Robert van Gent]

Assuming this is fixed, please reopen if not.

[Caleb Brown]

I finally got around to testing this. Unfortunately it doesn't work.

I suspect the issue is the creds used when there is no ADC is empty:

creds = &google.Credentials{}

So creds.TokenSource is not set and the API requests fail.

One possibility for solving this is using something like the following to generate a valid creds instance:

creds, _ = google.CredentialsFromJSON(ctx, []byte(`{"type": "service_account"}`))

Or use golang.org/x/oauth2/jwt to create the token source:

creds.TokenSource = &jwt.Config{
		Scopes:       []string{"https://www.googleapis.com/auth/cloud-platform"},
		TokenURL:     google.JWTTokenURL,
	}.TokenSource(ctx)

[Caleb Brown]

Thanks! I've tested this and it works now.

[Robert van Gent]

Oh thank goodness :-). Third time is a charm.