[QUESTION] Schemathesis only ran a single POST test and marked as success, when a failure would be expected

[pb-home]

Checklist

• I checked the FAQ section of the documentation
• I looked for similar issues in the issue tracker
• [x ] I am using the latest version of Schemathesis

Describe the bug

During positive testing against a simple POST that does not send a request body we receive back a 415 (unsupported media type). This test is marked as success and no further testing on the operation are carried out. The --checks switch was not used in the test run.

The mock was expecting a request body, responding with a 415.

The question is why only a single test which passed?
I assume that as no request body was in the specification, the test data was limited to a single test.
But why the pass?

To Reproduce

🚨 Mandatory 🚨: Steps to reproduce the behavior:

st run sampleapi.yaml --base-url http://localhost:8080/crawl --cassette-path sampleapi-results.yaml

1. Run this command '...'
2. See error

Please include a minimal API schema causing this issue:

openapi: 3.0.1
info:
  title: Website crawl for AIS POC
  contact:
    name: xxx
    url: xxx
    email: xxx
  license:
    name: Apache 2.0
    url: https://www.apache.org/licenses/LICENSE-2.0.html
  version: 0.0.1
tags:
- name: crawl
  description: Crawl endpoints
paths:
  /crawl:
    post:
      tags:
      - crawl
      summary: Creates a new crawl if one doesn't already exist. Returns information
        about the website crawl for the request
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                crawlResponse:
                  type: string
                  nullable: true

Environment

- OS: macOS
- Python version: 3.11.5
- Schemathesis version: 3.24.3
- Spec version: 3.0.1

Additional context

Attached cassette output
st-output.yaml.zip

[Dmitry Dygalo]

Hi @pb-own

I assume that as no request body was in the specification, the test data was limited to a single test.

You are right!

But why the pass?

I assume you'd expect Schemathesis tests to fail because 415 is not documented?

As you mentioned, the --checks option was not used hence only the not_a_server_error check was used which checks for 5xx status codes. The API returned 415, so the check passed.

Or its more a behavior mismatch that Schemathesis does not detect? I.e. the test is considered positive, but the API indicated a client error and it is not reported?

For the latter case I have this check implemented for my own usage, but I see it hard to detect such mismatches for an arbitrary case hence it is not included in the built-in checks. For example, Schemathesis may send some ID in the payload, but the API responds 404 (or 422, or 400, or something else), which is a reasonable response for an unknown ID, but its a positive test case from the spec perspective, even though the API indicates a client error.

[Dmitry Dygalo]

Checking if the issue is still relevant. Let me know if the comment above is helpful, otherwise I'd be happy to elaborate