mozilla / gcp-ingestion

Documentation and implementation of telemetry ingestion on Google Cloud Platform

Home Page:https://mozilla.github.io/gcp-ingestion/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add BIG_QUERY_JOB_PROJECT or similar configuration option to ingestion-sink

whd opened this issue · comments

commented

/CC @relud @acmiyaguchi

For pioneer, due to being within a service perimeter, we need batch load jobs to happen in the equivalent of shared-prod (i.e. the project where the bigquery data is homed). The easiest way to accomplish this is with an option to the sink.

Actually the easiest way is to set GOOGLE_CLOUD_PROJECT but in this case we only want the batch load jobs to happen in the shared-project, not e.g. metrics writing. When I enabled this in pioneer the sink can now write to bigquery at the expense of a bunch of com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Field timeSeries[0].resource.labels.project_id had an invalid value of "moz-fx-data-pioneer-prod-5e99": if present, must be the project number or ID in the request name. errors.