public-api-demos
Description
Demo to show how to use the Datafold API to run data diffs in CI pipelines in all kinds of ways.
Example 1: Running Datafold Cloud in a Jenkins Pipeline
Tech Stack
- dbt Cloud
- BigQuery
- Datafold
- python 3.9.6
- Jenkins (as reference)
Run Locally
# setup python venv
pip install --upgrade pip
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
source venv/bin/activate
# test in a personal repo, this example will not work automatically unless you have a GCP service account
git clone https://github.com/sungchun12/dbt_bigquery_example.git
cd dbt_bigquery_example
git checkout demo/bigquery-pr# add these Datafold configs to your `dbt_project.yml`
vars:
data_diff:
datasource_id: 6575 # found in the Datafold Data Sources Settings Page# export env vars for a local run, few examples below
# you'll inject these in the jenkins file, these are exported locally for testing
export DBT_CLOUD_PRODUCTION_JOB_ID=<your_dbt_cloud_job_id>
export DATAFOLD_API_KEY=<your_datafold_api_key>
export DBT_CLOUD_API_TOKEN=<your_dbt_cloud_api_token>
export DATA_SOURCE_ID=<your_data_source_id># run the script
python ../datafold_ci.py
# output example
Getting latest job run artifacts [manifest.json] for job id: 428271
File written to: /Users/sung/api-demo/dbt_bigquery_example/manifest.json
Getting latest job run artifacts [run_results.json] for job id: 428275
File written to: /Users/sung/api-demo/dbt_bigquery_example/target/run_results.json
Getting latest job run artifacts [manifest.json] for job id: 428275
File written to: /Users/sung/api-demo/dbt_bigquery_example/target/manifest.json
Running with data-diff=0.8.4 (Update 0.9.1 is available!)
Cloud datafold host: https://app.datafold.com
Saving the API key to the system keyring service
Diffs in progress...
dbt-demo-386220.dbt_sung_prod.order_payments <>
dbt-demo-386220.dbt_cloud_pr_ALPHANUMERIC_CHANGE_BRANCH_CHANGE_ID_BUILD_NUMBER_428275.order_payments
Skipped due to unknown primary key. Add uniqueness tests, meta, or tags.
fct_orders: https://app.datafold.com/datadiffs/2262836/overview
stg_orders: https://app.datafold.com/datadiffs/2262837/overview
stg_payments: https://app.datafold.com/datadiffs/2262838/overview
dbt-demo-386220.dbt_sung_prod.fct_orders <>
dbt-demo-386220.dbt_cloud_pr_ALPHANUMERIC_CHANGE_BRANCH_CHANGE_ID_BUILD_NUMBER_428275.fct_orders
Column(s) added: ['new_column']
https://app.datafold.com/datadiffs/2262836/overview
Rows Added Rows Removed
------------ --------------
10 0
Updated Rows: 0
Unchanged Rows: 10
Value Match Percent:
dbt-demo-386220.dbt_sung_prod.stg_orders <>
dbt-demo-386220.dbt_cloud_pr_ALPHANUMERIC_CHANGE_BRANCH_CHANGE_ID_BUILD_NUMBER_428275.stg_orders
https://app.datafold.com/datadiffs/2262837/overview
No row differences
dbt-demo-386220.dbt_sung_prod.stg_payments <>
dbt-demo-386220.dbt_cloud_pr_ALPHANUMERIC_CHANGE_BRANCH_CHANGE_ID_BUILD_NUMBER_428275.stg_payments
https://app.datafold.com/datadiffs/2262838/overview
No row differences