[BUG] hypothesis.errors.InvalidDefinition: Type APIWorkflow defines no rules

[jakub-krysl-ipf]

Checklist

• I checked the FAQ section of the documentation
• I looked for similar issues in the issue tracker
• I am using the latest version of Schemathesis

Describe the bug

Following docs for Lazy schema loading I cannot get the state_machine to run. I simplified it down to following code:

import schemathesis
from pytest import fixture


@fixture
def state_machine():
    schema = schemathesis.from_dict({
        "openapi": "3.0.2",
        "info": {},
        "paths": {}
    })
    return schema.as_state_machine()


def test_statefully(state_machine):
    state_machine.run()

______________________________________________________________________________________________________________ test_statefully _______________________________________________________________________________________________________________

state_machine = <class 'schemathesis.specs.openapi.stateful.APIWorkflow'>

    def test_statefully(state_machine):
>       state_machine.run()

tests/api/test_schemathesis.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv/lib/python3.11/site-packages/schemathesis/stateful/state_machine.py:60: in run
    return run_state_machine_as_test(cls, settings=settings)
venv/lib/python3.11/site-packages/schemathesis/stateful/__init__.py:133: in run_state_machine_as_test
    return _run_state_machine_as_test(state_machine_factory, settings=settings, _min_steps=2)
venv/lib/python3.11/site-packages/hypothesis/stateful.py:217: in run_state_machine_as_test
    run_state_machine(state_machine_factory)
venv/lib/python3.11/site-packages/hypothesis/stateful.py:113: in run_state_machine
    @given(st.data())
venv/lib/python3.11/site-packages/schemathesis/stateful/state_machine.py:41: in __init__
    super().__init__()  # type: ignore
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = APIWorkflow({})

    def __init__(self) -> None:
        if not self.rules():
>           raise InvalidDefinition(f"Type {type(self).__name__} defines no rules")
E           hypothesis.errors.InvalidDefinition: Type APIWorkflow defines no rules
E           Falsifying example:

venv/lib/python3.11/site-packages/hypothesis/stateful.py:248: InvalidDefinition

To Reproduce

🚨 Mandatory 🚨: Steps to reproduce the behavior:

1. create a test file with above code
2. pytest -k test_statefully
3. See error

Please include a minimal API schema causing this issue:

{
        "openapi": "3.0.2",
        "info": {},
        "paths": {}
}

Expected behavior

Able to run state_machine as documented.

Environment

- OS: Fedora 39
- Python version: 3.11 venv
- Schemathesis version: 3.24.3
- Spec version: Open API 3.0.2

Additional context

Originally I used real OpenApi schema which I am able to lazily load using from_pytest_fixture() loading using schemathesis.from_uri(), but the issue does not seem to be in the schema itself. I tried running with state_machine.run(settings=settings(max_examples=1)) but that does not help.

[Dmitry Dygalo]

Hi @jakub-krysl-ipf

Does the actual schema contain any Open API links?

[jakub-krysl-ipf]

@Stranger6667 I don't think I understand the reason behind your question. The actual schema is quite complex and to be honest really irrelevant for the bug. I ran the code above in the description with the dummy schema and it produced the same result.

So for the reproducer it does not contain any links (or any real data) as the whole schema needed to hit the bug is just this:

{
        "openapi": "3.0.2",
        "info": {},
        "paths": {}
}

[Dmitry Dygalo]

The reason for the question is that stateful testing requires Open API links to be defined to run stateful tests:

To specify how different operations depend on each other, we use a special syntax from the Open API specification - Open API links

Perhaps it does not clearly state that running stateful tests requires specifying at least one connection between two API operations (though it is briefly mentioned in the How it works behind the scenes section).

Similarly, the error message is not tuned to reflect the API testing aspect. It comes from the underlying framework and merely reflects the implementation detail.

Other than that, the intention behind the error message is to show there is no point in running stateful tests if there are no stateful transitions in the schema. I.e. such config would be equivalent to the regular tests generated by schema.parametrize, but in a randomized order.

Hopefully, if the error message would reflect this aspect and provide some actionable guidance on how to setup stateful tests, then it would be easier to use.

Do these details clarify the reason behind the actual behavior? If you have any suggestions or more questions, I'd be happy to discuss them.

[jakub-krysl-ipf]

@Stranger6667 Thanks for the explanation. My understanding from the docs was that if there are no links for cases, they are generated same as stateless. So if there are no links at all, the test set will be same as stateless except in random order.
The error Type APIWorkflow defines no rules made me think I did not specify settings, it did not point me to "you have no links in you Open API schema". It would be great if you can make the error clearer with guidelines. 👍

I think I owe you an explanation why I'm trying to run stateful tests without links.

I hit an issue with current implementation of stateless tests, where originally we use lazy loading to postpone the schema load. URL to API contains version in it and it is not know at the time we load module. We've had a workaround to manually specify it using env var, but I rewrote it to remove this and actually check the version on the server.

Which means I need the server, which is available after running a fixture.

But to be able to use it for schemathesis I need to move the schema loading to fixture too. And I did not find a way to do this with stateless tests, but stateful support this. Is there a reason behind this?

So I cannot easily use pytest.mark.skipif based on version since fixtures run after that. I created a custom mark defined by fixture that provides the version. For now I guess I'll have to move the server fixture to collection hook (ughhh) to have the server available before schemathesis lazy loading is called (when decorators are called). This has a benefit of not having to work around pytest.mark.skipif, but I'll have to move cleanup to some hook called after test execution. Not as clean as fixture...

[Dmitry Dygalo]

Thank you for providing the context!

My understanding from the docs was that if there are no links for cases, they are generated same as stateless.

I changed this behavior aspect some time ago, but forgot to update the docs accordingly. Will work on the clarifications, so they will be shipped in the next release

But to be able to use it for schemathesis I need to move the schema loading to fixture too. And I did not find a way to do this with stateless tests, but stateful support this. Is there a reason behind this?

I think that lazy loading should work:

from contextlib import asynccontextmanager
from fastapi import FastAPI
import pytest
import schemathesis


@pytest.fixture
def web_app(db):
    # some dynamically built application
    # that depends on other fixtures
    app = FastAPI()

    @asynccontextmanager
    async def lifespan(_: FastAPI):
        await db.connect()
        yield
        await db.disconnect()

    return schemathesis.from_dict(app.openapi())


schema = schemathesis.from_pytest_fixture("web_app")


@schema.parametrize()
def test_api(case):
    ...

In this example, the schema is loaded inside the web_app fixture - is it something you are looking for?

So I cannot easily use pytest.mark.skipif based on version since fixtures run after that. I created a custom mark defined by fixture that provides the version. For now I guess I'll have to move the server fixture to collection hook (ughhh) to have the server available before schemathesis lazy loading is called (when decorators are called). This has a benefit of not having to work around pytest.mark.skipif, but I'll have to move cleanup to some hook called after test execution. Not as clean as fixture...

Interesting! Maybe it could be that the mark is propagated to the fixture, which loads the server (it should be available via the request.node.get_closest_marker("pytest.mark.skipif")) and as the version is loaded, then check it and call pytest.skip if needed?

[jakub-krysl-ipf]

I changed this behavior aspect some time ago, but forgot to update the docs accordingly. Will work on the clarifications, so they will be shipped in the next release

Thanks!

I think that lazy loading should work:

Sadly we already have this and it does not work for our case. It seems all decorators are calculated before they are applied. So decorator order does not matter and I cannot first skip and than load schema. To use lazy loading with version check I need to add 2nd version check to the web_app fixture.

Interesting! Maybe it could be that the mark is propagated to the fixture, which loads the server (it should be available via the request.node.get_closest_marker("pytest.mark.skipif")) and as the version is loaded, then check it and call pytest.skip if needed?

Interesting approach, I did not think of using the skipif marker itself as trigger to load the version variable. This way I can make the code that triggers version load to be called only when the skipif is encountered. It narrows it a little, but assumes we don't have other skipif than those based on version. Plus the code would probably have to be at the same place as now - the collection hook. Having it in fixture as decorator still evaluates the lazy loaded schema.