mozilla / bigquery-etl

Bigquery ETL

Home Page:https://mozilla.github.io/bigquery-etl

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Adding the ability to override scheduling parameters on backfill

braunreyes opened this issue · comments

Currently the query backfill command only supports {{ds}} as a submission_date:DATE: scheduling parameter. However, it is possible for folks to add more complex logic for the normal bqetl runs.

For example: https://github.com/mozilla/private-bigquery-etl/blob/56190e888f051c7190241767d0b7519665bc70c9/sql/moz-fx-data-shared-prod/ads_derived/tiles_addressable_inventory_hourly_v1/metadata.yaml#L22

We need to backfill this table for a fill range of days, which means we do not need this custom logic for submission date. Given the way it is currently written, it seems as though providing the ability to completely override the list of parameters at backfill runtime is the best solution.

If the new flag is used, then scheduling parameters will be completely overridden. This will not be a 'merge' of parameters.

┆Issue is synchronized with this Jira Task