[BUG] hypothesis.errors.InvalidArgument: test has already been decorated with a settings object.
dkbarn opened this issue · comments
Checklist
- [ X ] I checked the FAQ section of the documentation
- [ X ] I looked for similar issues in the issue tracker
- [ X ] I am using the latest version of Schemathesis
Describe the bug
After upgrading from schemathesis 3.19.5 to 3.26.0, several tests are now failing with this error message:
hypothesis.errors.InvalidArgument: test_graphql_schema has already been decorated with a settings object.
To Reproduce
Here is an example test that produces the above error:
import copy
import falcon
import hypothesis
from hypothesis import HealthCheck
import pytest
import schemathesis
from .util import GRAPHQL_HEADERS, create_database
def graphql_schema_from_asgi(db_setup_factory):
setup = db_setup_factory()
# this slows things down but we need this fixture to not be async
# so schemathesis.from_pytest_fixture can use it correctly
falcon.async_to_sync(create_database, setup.sql_engine_manager)
endpoint = "/v1/graphql"
return schemathesis.graphql.from_asgi(endpoint, setup.app, headers=GRAPHQL_HEADERS)
graphql_schema = schemathesis.from_pytest_fixture("graphql_schema_from_asgi")
@graphql_schema.parametrize()
@hypothesis.settings(
deadline=None,
suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much],
)
def test_graphql_schema(case):
response = case.call_asgi()
case.validate_response(response)The problem seems to be the fact that we are decorating our test with both the @graphql_schema decorator generated by schemathesis and the @hypothesis.settings decorator. But we need to do this, in order to customize the behavior of hypothesis for our test.
Expected behavior
The expected behaviour is that a test which passes without issue in 3.19.5 of schemathesis would continue to work in 3.26.0
Environment
- OS: EL7
- Python version: 3.8
- Schemathesis version: 3.26.0
Additional context
Hi! This is weird, I think we should have tests for it.
What is your pytest version?
And pytest-subtests version would be useful to know too
Sure thing, the test venv has:
pytest 8.1.1
pytest-subtests 0.7.0
(venv) bash-4.2$ poetry show pytest
name : pytest
version : 8.1.1
description : pytest: simple powerful testing with Python
dependencies
- colorama *
- exceptiongroup >=1.0.0rc8
- iniconfig *
- packaging *
- pluggy >=1.4,<2.0
- tomli >=1
required by
- pytest-asyncio >=7.0.0
- pytest-benchmark >=3.8
- pytest-cov >=4.6
- pytest-datadir >=5.0
- pytest-mock >=6.2.5
- pytest-regressions >=6.2.0
- pytest-subtests >=7.0
- schemathesis >=4.6.4,<9
(venv) bash-4.2$ poetry show pytest-subtests
name : pytest-subtests
version : 0.7.0
description : unittest subTest() support and subtests fixture
dependencies
- pytest >=7.0
required by
- schemathesis >=0.2.1,<0.8.0
I tried to reproduce it with a simpler version but wasn't successful. I also noted that graphql_schema_from_asgi has no @pytest.fixture decorator - its a typo in the original comment, right?
Generally, I don't expect that the exact Python version / Platform would matter here, but will take a closer look.
import hypothesis
import pytest
from hypothesis import HealthCheck
import schemathesis
from test.apps import _graphql
@pytest.fixture
def graphql_schema():
app = _graphql._fastapi.create_app()
return schemathesis.graphql.from_asgi("/graphql", app)
schema = schemathesis.from_pytest_fixture("graphql_schema")
@schema.parametrize()
@hypothesis.settings(
deadline=None,
suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much],
)
def test_graphql_schema(case):
response = case.call_asgi()
case.validate_response(response)❯ pytest sample.py
======================================= test session starts =======================================
platform darwin -- Python 3.11.7, pytest-8.1.1, pluggy-1.4.0
codspeed: 2.2.0 (callgraph: not supported)
rootdir: /tmp/schemathesis
configfile: pyproject.toml
plugins: syrupy-4.6.1, schemathesis-3.26.0, codspeed-2.2.0, hypothesis-6.98.10, mock-3.10.0, asyncio-0.21.1, trio-0.8.0, doctestplus-0.12.1, anyio-3.7.1, xdist-3.3.1, forked-1.6.0, subtests-0.7.0, httpserver-1.0.6
asyncio: mode=Mode.STRICT
collected 1 item
sample.py ,,,,.
======================================== 1 passed in 3.24s ========================================
How do you run your tests? Maybe there is some CLI arg, that could interfere?
You're right, I had left out a key detail from my example code which it turns out is the cause of the error. It seems that the error happens when the schema fixture is parametrized. Here is an example to demonstrate:
import copy
import falcon
import hypothesis
from hypothesis import HealthCheck
import itertools
import pytest
import schemathesis
from .util import GRAPHQL_HEADERS, create_database
INTERFACES = ["v1"]
DB_SCHEMAS = ["db0"]
@pytest.fixture(params=itertools.product(INTERFACES, DB_SCHEMAS))
def graphql_schema_from_asgi(request, db_setup_factory):
interface, db_schema = request.param
setup = db_setup_factory(schema=db_schema)
# this slows things down but we need this fixture to not be async
# so schemathesis.from_pytest_fixture can use it correctly
falcon.async_to_sync(create_database, setup.sql_engine_manager)
endpoint = f"/{interface}/graphql"
return schemathesis.graphql.from_asgi(endpoint, setup.app, headers=GRAPHQL_HEADERS)
graphql_schema = schemathesis.from_pytest_fixture("graphql_schema_from_asgi")
@graphql_schema.parametrize()
@hypothesis.settings(
deadline=None,
suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much],
)
def test_graphql_schema(case):
response = case.call_asgi()
case.validate_response(response)Thank you! I can reproduce it now.
Looks like it starts failing if Hypothesis is >=6.84.3, I'll fix it soon
Fixed in 3.26.1
Confirmed that the error is now resolved with 3.26.1 - thanks for the super fast turnaround!
Awesome! Happy to help