`compare_all_columns` fails if table has too many columns in BigQuery

[Bruno Kim Medeiros Cesar]

Describe the bug

When trying to apply audit_helper.compare_all_columns to a table with 196 columns, the test fails with

18:07:33    Resources exceeded during query execution: Not enough resources for query planning - too many subqueries or query is too complex.
18:07:33    compiled SQL at target/run/****/src/tests/****_unit_test.sql

Note that the compared tables have a single row each, as I'm trying to use audit_helper for a SQL unit test.

This is more of a BigQuery limitation then a bug with the library. However, it may be possible to circumvent it if somehow we can paginate over all columns.

Steps to reproduce

1. Create a table with 196 columns. Their types probably don't matter.
2. Create a second table with the same schema.
3. Try to run audit_helper.compare_all_columns comparing them.

Expected results

I'd like there to be a report showing which columns don't match among the rows.

Actual results

The query fails because it's too complex

Screenshots and log output

N/A (sensitive information)

System information

The contents of your packages.yml file:

packages:
  - package: dbt-labs/dbt_utils
    version: 0.9.0
  - package: dbt-labs/audit_helper
    version: 0.6.0

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

Core:
  - installed: 1.2.3
  - latest:    1.3.1 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - bigquery: 1.2.0 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

The operating system you're using:
Linux 5.15.0-56-generic #62~20.04.1-Ubuntu SMP Tue Nov 22 21:24:20 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

The output of python --version:
Python 3.7.13

Additional context

Are you interested in contributing the fix?

I may contribute with guidance, though I haven't signed any agreement

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.