[Bug] Projects with `restrict-access: true` cannot leverage generic tests on private models
nicholasyager opened this issue · comments
Is this a new bug in dbt-core?
- I believe this is a new bug in dbt-core
- I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
Suppose you have a dbt project with restrict-access: true
, preventing other projects from importing this project and leveraging protected models. Now suppose you have a model with a group
defined and access
set to private
, and a column with a generic test -- let's say a not_null
test for the sake of example. With this combination of settings in place, compilation will raise a DbtReferenceError
on the test since the test itself is not part of the group
.
__models.yml
models:
- name: accounts
description: >
All accounts with whom we have done business. This is a very sensitive asset.
access: private
group: sales
columns:
- name: name
description: Name of the account.
tests:
- not_null
- unique
dbt_project.yml
...
restrict-access: true
...
Output
(venv-1.7) > $ dbt build
00:22:16 Running with dbt=1.7.14
00:22:17 Registered adapter: duckdb=1.7.5
00:22:17 Encountered an error:
Parsing Error
Node test.revenue.not_null_accounts_name.a22fd7c4da attempted to reference node model.revenue.accounts, which is not allowed because the referenced node is private to the 'sales' group.
The specific defect can be resolved in manifest.py
in the Manifest.is_invalid_private_ref
method. This method should be updated to valuate to false
then the calling node
's resource_type
is NodeType.Test
.
Expected Behavior
I would expect a dbt test Node to not be beholden to private
ref checks when derived from a generic test.
Steps To Reproduce
See description
Relevant log output
(venv-1.7) > $ dbt build
00:22:16 Running with dbt=1.7.14
00:22:17 Registered adapter: duckdb=1.7.5
00:22:17 Encountered an error:
Parsing Error
Node test.revenue.not_null_accounts_name.a22fd7c4da attempted to reference node model.revenue.accounts, which is not allowed because the referenced node is private to the 'sales' group.
Environment
- OS: macos 14.4.1
- Python: 3.12.3
- dbt: 1.7.14 and 1.8.0
Which database adapter are you using with dbt?
snowflake, other (mention it in "Additional Context")
Additional Context
Specific cause of the defect is here:
dbt-core/core/dbt/contracts/graph/manifest.py
Line 1434 in 8fe7d65
This issue is related to #8248
Thanks for opening @nicholasyager!
I agree that this is a bug. But I think it's slightly different from #8248, because generic tests can belong to groups
, and dbt automatically sets their group
attribute to match the model they're defined on.
The current intended behavior is:
- If a model is installed via a "package" dependency, and
restrict-access
isFalse
for that package, then it should be treated exactly the same as resources within this project. (This maintains historical behavior for packages.) - If a model is installed via a "package" dependency, and
restrict-access
isTrue
, then the difference in namespace (package_name
) should restrict private and protected references. - If a model is installed as a "project" dependency, then only reference to
public
models is allowed. To that end, theaccess
level and namespace (package_name
) are sufficient for resolving that reference, asgroups
don't currently exist across projects.
The current issue is that dbt's application of (2) is too broad, applying restrict-access
even to resources within the same package/group. Rather than changing this logic to skip private reference checks for tests, I think we need to change this logic to be something like the following:
def is_invalid_private_ref(
...
return is_private_ref and (
# Invalid reference because the group does not match
(hasattr(node, "group") and node.group and node.group != target_model.group)
# Or, invalid because these are different namespaces (project/package) and restrict-access is enforced
or (node.package_name != target_model.package_name and restrict_package_access)
)
When I try that change locally, and your reproduction case, dbt passes (and does not raise a DbtReferenceError
).
Speaking of groups across projects...
Feature: Support cross-project Group evaluation
It sounds like you might be thinking in a direction similar to this proposal:
The idea there is, a model should be ref'able by specific other projects, based on some opt-in criteria shared by both. If we pursue that idea, it would make sense to add group
(and whatever other necessary properties) to ModelNodeArgs
(which I see came up in linked conversation).
Open to hearing your thoughts on #9340!