TFX on Cloud AI Platform Pipelines: TFX pipeline run cannot write into defined bucket (403 Insufficient Permission)
jpatokal opened this issue · comments
In Lab 02, when the TFX Tuner SA is configured as documented:
CUSTOM_SERVICE_ACCOUNT = 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-1057c4de4b13.iam.gserviceaccount.com'
Runs of the TFX pipeline fail because Pipelines can't write into the bucket (which is missing by default, see issue #124).
tensorflow.python.framework.errors_impl.PermissionDeniedError: Error executing an HTTP request: HTTP response code 403 with body '{
"error": {
"code": 403,
"message": "Insufficient Permission",
"errors": [
{
"message": "Insufficient Permission",
"domain": "global",
"reason": "insufficientPermissions"
}
]
}
}
'
when initiating an upload to gs://my-missing-bucket/tfx_covertype_continuous_training/
I'm somewhat baffled as to why, since the Tuner SA and a few more all have Object Storage Admin privs on the bucket:
qwiklabs-gcp-01-1057c4de4b13@qwiklabs-gcp-01-1057c4de4b13.iam.gserviceaccount.com | Qwiklabs User Service Account | Storage Admin | qwiklabs-gcp-01-1057c4de4b13 | |
service-306357166946@cloud-ml.google.com.iam.gserviceaccount.com | Google Cloud ML Engine Service Agent | AI Platform Service AgentStorage Object Admin | qwiklabs-gcp-01-1057c4de4b13 qwiklabs-gcp-01-1057c4de4b13
tfx-tuner-caip-service-account@qwiklabs-gcp-01-1057c4de4b13.iam.gserviceaccount.com | TFX Tuner CAIP Vizier | Storage Object Admin
Unfortunately you can't really tell from the logs which SA it's using.