[BUG] Schemathesis fails with pytest integration but works with CLI

[𝕂']

Checklist

• I checked the FAQ section of the documentation
• I looked for similar issues in the issue tracker
• I am using the latest version of Schemathesis

Describe the bug

When running Schemathesis via the CLI command (st run app/openapi.yaml --base-url=http://localhost:8000), it works as expected:

# st run app/openapi.yaml --base-url=http://localhost:8000

===================================================================================================== Schemathesis test session starts ====================================================================================================
Schema location: file:///workspace/app/openapi.yaml
Base URL: http://localhost:8000
Specification version: Open API 3.0.3
Random seed: 73278962497964048884064961632552961370
Workers: 1
Collected API operations: 5
Collected API links: 0
API probing: SUCCESS
Schema analysis: SKIP

GET /users              [ 20%]
POST /users             [ 40%]
GET /users/{user_id}    [ 60%]
PUT /users/{user_id}    [ 80%]
DELETE /users/{user_id} [100%]

================================================================================================================= SUMMARY =================================================================================================================
Performed checks:
    not_a_server_error                    401 / 401 passed          PASSED

Tip: Use the `--report` CLI option to visualize test results via Schemathesis.io.
     We run additional conformance checks on reports from public repos.

However, when running Schemathesis with pytest using the following test suite:

import schemathesis

schema = schemathesis.from_uri("http://localhost:8000/openapi.json")

@schema.parametrize()
def test_api(case):
    case.call_and_validate()

The test fails with the following error:

# pytest app/test_schemathesis.py

=========================================================================================================== test session starts ===========================================================================================================
platform linux -- Python 3.11.9, pytest-8.2.0, pluggy-1.5.0
rootdir: /workspace
configfile: pyproject.toml
plugins: hypothesis-6.100.5, anyio-4.3.0, subtests-0.7.0, schemathesis-3.27.1
collected 0 items / 1 error

================================================================================================================= ERRORS ==================================================================================================================
__________________________________________________________________ ERROR collecting app/test_schemathesis.py __________________________________________________________________
app/test_schemathesis.py:3: in <module>
    schema = schemathesis.from_uri("http://localhost:8000/openapi.json")
/usr/local/lib/python3.11/site-packages/schemathesis/specs/openapi/loaders.py:165: in from_uri
    return from_file(
/usr/local/lib/python3.11/site-packages/schemathesis/specs/openapi/loaders.py:256: in from_file
    return from_dict(
/usr/local/lib/python3.11/site-packages/schemathesis/specs/openapi/loaders.py:306: in from_dict
    from .schemas import OpenApi30, SwaggerV20
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/usr/local/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
/usr/local/lib/python3.11/site-packages/schemathesis/specs/openapi/schemas.py:85: in <module>
    SCHEMA_PARSING_ERRORS = (KeyError, AttributeError, jsonschema.exceptions.RefResolutionError)
/usr/local/lib/python3.11/site-packages/jsonschema/exceptions.py:30: in getattr
    warnings.warn(
E   DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.

========================================================================================================= short test summary info =========================================================================================================
ERROR app/test_schemathesis.py - DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

To Reproduce

🚨 Mandatory 🚨: Steps to reproduce the behavior:

1. Set up a project with an OpenAPI specification file.
2. Install Schemathesis and pytest.
3. Create a test file with the following content:

# test_schemathesis.py
import schemathesis

schema = schemathesis.from_uri("http://localhost:8000/openapi.json")

@schema.parametrize()
def test_api(case):
    case.call_and_validate()

4. Run the test using the command pytest test_schemathesis.py.

Please include a minimal API schema causing this issue:

I will attach this as a zip file in the Additional context section.

Expected behavior

The pytest should work without errors.

Environment

- OS: Linux(Debian GNU/Linux 12 (bookworm))
- Python version: 3.11.9
- Schemathesis version: 3.27.1
- Spec version: Open API 3.0.3
- FastAPI version: 0.110.1
- uvicorn version: 0.29.0

In my environment, I am using FastAPI (version 0.110.1) and Uvicorn (version 0.29.0) to serve the API.

from pathlib import Path
from fastapi import FastAPI
import yaml

app = FastAPI(debug=True, docs_url="/docs/users")

oas_doc = yaml.safe_load(
    (Path(__file__).parent / "./openapi.yaml").read_text()
)
app.openapi = lambda: oas_doc

Additional context

• The issue occurs when using the pytest integration, but not when running Schemathesis via the CLI.
• The deprecation warning is related to the jsonschema package used internally by Schemathesis. However, I suspect that the root cause of the issue may be something else, as the deprecation warning from jsonschema appears to be a side effect rather than the primary problem.

Please let me know if you need any further information or if there's anything else I can provide to help investigate this issue. I'll be happy to provide additional details or run any suggested tests to help identify the root cause of the problem.

openapi.yaml.zip

[Dmitry Dygalo]

Hi!

Thank you for the detailed report!

The deprecation warning is related to the jsonschema package used internally by Schemathesis. However, I suspect that the root cause of the issue may be something else, as the deprecation warning from jsonschema appears to be a side effect rather than the primary problem.

I think it has everything to do with HOW you run pytest and its configuration. There is the -W configuration option that makes pytest crash on certain warnings and I suspect that it is configured somewhere, as it is not the default behavior (e.g. in pyproject.toml, pytest.ini, etc).

Could you, please, share the configuration file you have if any?
I.e. there are tests that involve exactly the same steps as in the reproduction code + it does not reproduce on a fresh project:

Though it fails with -W error

[𝕂']

@Stranger6667
Hi!

Thank you for your quick response and insightful feedback.

You are absolutely correct.
After receiving your response, I double-checked my pyproject.toml file and found the following configuration:

filterwarnings = [
    "error",
]

The "error" setting in the filterwarnings was causing pytest to treat warnings as errors. After removing the "error" line, the tests passed successfully.

I apologize for not thoroughly verifying the issue before submitting the bug report.
The problem was indeed related to my pytest configuration and not a bug in Schemathesis itself.

Thank you for taking the time to review the report and pointing me in the right direction.
I appreciate your help and the effort you put into maintaining this project

Please feel free to close this issue as it has been resolved.
Best regards,

[Dmitry Dygalo]

@K-dash no worries! I appreciate you opening an issue to solve this! :) Please, feel free to reach out to me if you have any further questions.