Deployment of Beam pipeline not working in GCB
jianglai opened this issue · comments
Sample GCB job output:
It is not clear to me why. The service account was granted both GAE admin and storage admin role. Although if I run each command locally, I can deploy the pipelines fine, with the following warning:
$ java -jar nomulus.jar -e production --credential tool-credential.json deploy_invoicing_pipeline
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Jul 11, 2019 2:36:46 PM com.google.auth.oauth2.DefaultCredentialsProvider warnAboutProblematicCredentials
WARNING: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/.
Dataflow SDK version: 2.11.0
It looks like even though tool-credential.json (a service account credential) is supplied, the Beam SDK still uses end user credentials created by gcloud auth login. I wonder if the recent change in credential handling has some inadvertent effect?
I see, the weird thing is we removed the credential for invoicing pipeline for serialization problem but we didn't make any change to spec11 pipeline, but it also failed with the same error based on the log information. I will take a close look to how the credential is used.
Also, here is the successful deployment I had the first time I added the deployment job, it still used the deprecated credential at that time. You can see there is no such "WARNING: Your application has authenticated using end user credentials" in the log.
I checked the spec11 pipeline code(which was not changed recently) and I believe we never tried to use the provided credential to deploy it, instead, we rely on ADC to find the credential. Previously, we always manually deploy the pipeline from our desktop where user credential is installed and has permission to all GCP projects we own so it is not a problem. Also, the previous test I did was to deploy the pipeline from alpha's Cloud Build to alpha's GCS, so Cloud Build's host should have the credential to do all the work.
However, the failed production build was to deploy the pipeline from dev account's Cloud Build to production account's GCS, so there was no valid credential unless we explicitly give it to the pipeline.
OK so the credential passed in --credential is not actually used to perform the deployment?
I don't think it is ever used for spec11 pipeline. It may be used by invoicing pipeline before we removed options.setGcpCredential(credentials) due to serialization problem.
I am going to add it back and test 1) if the warning disappears in local deployment 2) if a cross project deployment can succeed.