At end of job with load_method as REPLACE and target table is non partitioned, the target table schema gets updated as select query

[Sumit Agrawal]

The job instance with load method as REPLACE and the target table is non partitioned, while executing this specific instance of job, we are experiencing that the target table schema is getting updated as schema from the select query.

This behaviour is causing problem with data governance flow, as now the schema in BQ table does not match with schema from the resource.yaml file in ocean (git repository to host the table definations) repo.

[Sumit Agrawal]

The Approach to fix this issue is as below

Only in case of load_method as REPLACE and table is non partitioned

1. execute the query to truncate this table
2. execute the job same as previous flow but update the write_disposition property as WRITE_APPEND

At end of this execution we will achive the replace behaviour without re-creating the target table.