Add BIG_QUERY_JOB_PROJECT or similar configuration option to ingestion-sink
whd opened this issue · comments
/CC @relud @acmiyaguchi
For pioneer, due to being within a service perimeter, we need batch load jobs to happen in the equivalent of shared-prod (i.e. the project where the bigquery data is homed). The easiest way to accomplish this is with an option to the sink.
Actually the easiest way is to set GOOGLE_CLOUD_PROJECT
but in this case we only want the batch load jobs to happen in the shared-project, not e.g. metrics writing. When I enabled this in pioneer the sink can now write to bigquery at the expense of a bunch of com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Field timeSeries[0].resource.labels.project_id had an invalid value of "moz-fx-data-pioneer-prod-5e99": if present, must be the project number or ID in the request name.
errors.