[BUG] hypothesis-max-examples is ignored when using docker run

[dbire]

Checklist

• [x ] I checked the FAQ section of the documentation
• [ x] I looked for similar issues in the issue tracker
• [ x] I am using the latest version of Schemathesis

Describe the bug

Clearly describe the issue you're facing.

To Reproduce

🚨 Mandatory 🚨: Steps to reproduce the behavior:

1. Run this command
   'docker run --network=my-network
   -v /home/didier/projects/ck_api_design:/app
   schemathesis/schemathesis:stable run /app/api/v4/users.yaml
   --code-sample-style curl
   --workers auto
   --checks all
   --base-url http://qa-users-api:50103/api/v4
   --header "X-Api-Key: xxxxxxxxxx"
   --validate-schema=true
   --endpoint ^/api/v4/users$
   --method POST
   --hypothesis-max-examples 1'
2. See error
   After some minutes of execution (instead of a matter of seconds expected) we get:

=================================== SUMMARY ====================================

Performed checks:
not_a_server_error 5677 / 5677 passed PASSED
status_code_conformance 5674 / 5677 passed FAILED
content_type_conformance 5677 / 5677 passed PASSED
response_headers_conformance 5677 / 5677 passed PASSED
response_schema_conformance 3 / 5677 passed FAILED

If I use a previous version of schemathesis (v3.17.0):

Execution is a matter of seconds and the hypothesis-max-examples option works as expected:

Performed checks:
not_a_server_error 7 / 7 passed PASSED
status_code_conformance 4 / 7 passed FAILED
content_type_conformance 7 / 7 passed PASSED
response_headers_conformance 7 / 7 passed PASSED
response_schema_conformance 4 / 7 passed FAILED

Please include a minimal API schema causing this issue:

{ "openapi": "3.0.2", ... }

Expected behavior

Clearly describe your expected outcome.

Environment

- Linux
- Python version: Depends on schemathesis image used in stable
- Schemathesis version: [e.g. 3.20.0] 3.21.0
- Spec version: [e.g. Open API 3.0.2]  3.0.1

Additional context

Include any other relevant details, like logs or screenshots.

[dbire]

After some testing I found out that the regression was introduced in v3.20.0 since v3.19.0 works as expected as regards hypothesis-max-examples

[Dmitry Dygalo]

Hi @dbire !

Thanks for reporting! Does it work in a non-Docker environment?

[dbire]

Hi @Stranger6667 ! Yes, I do confirm it works as expected in a non-Docker environment

[Dmitry Dygalo]

Unfortunately, I am not able to reproduce it with v3.20.0 / v3.21.2:

❯ docker run -v /Users/dmitrydygalo/CLionProjects/schemathesis:/app schemathesis/schemathesis:v3.20.0 run --checks all schema.json --request-tls-verify=false --force-color --validate-schema=true --workers auto --endpoint response-conformance --hypothesis-max-examples 1 --base-url=http://example.schemathesis.io --code-sample-style curl --method GET --header "X-Api-Key: xxxxxxxxxx"
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
======================= Schemathesis test session starts =======================
Schema location: file:///app/schema.json
Base URL: http://example.schemathesis.io
Specification version: Open API 3.0.2
Workers: 5
Collected API operations: 4

.FFF

=================================== FAILURES ===================================
___________________ GET /response-conformance/malformed-json ___________________
1. The received response is not valid JSON:

    {success: true}

Exception: 

    Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

Response status: 200
Response payload: `{success: true}`

Run this cURL command to reproduce this failure: 

    curl -X GET -H 'X-Api-Key: [Filtered]' --insecure http://example.schemathesis.io/response-conformance/malformed-json

X-Schemathesis-TestCaseId: XQhOlb

Or add this option to your command line parameters: --hypothesis-seed=110468662907739167066722197348536649600
_______________ GET /response-conformance/incorrect-content-type _______________
1. Received a response with 'text/plain' Content-Type, but it is not declared in the schema.

Defined content types: application/json

Response status: 200
Response payload: `Success!`

Run this cURL command to reproduce this failure: 

    curl -X GET -H 'X-Api-Key: [Filtered]' --insecure http://example.schemathesis.io/response-conformance/incorrect-content-type

X-Schemathesis-TestCaseId: mViQtX

Or add this option to your command line parameters: --hypothesis-seed=305714469661256543834562175878995455096
___________________ GET /response-conformance/missing-field ____________________
1. The received response does not conform to the defined schema!

Details: 

'age' is a required property

Failed validating 'required' in schema

Response status: 200
Response payload: `{"id":"123","name":"Alice"}
`

Run this cURL command to reproduce this failure: 

    curl -X GET -H 'X-Api-Key: [Filtered]' --insecure http://example.schemathesis.io/response-conformance/missing-field

X-Schemathesis-TestCaseId: QtkyAW

Or add this option to your command line parameters: --hypothesis-seed=195100063524251545362187029848178182008
=================================== SUMMARY ====================================

Performed checks:
    not_a_server_error                              7 / 7 passed          PASSED 
    status_code_conformance                         7 / 7 passed          PASSED 
    content_type_conformance                        5 / 7 passed          FAILED 
    response_headers_conformance                    7 / 7 passed          PASSED 
    response_schema_conformance                     3 / 7 passed          FAILED 

Note: The 'X-Schemathesis-TestCaseId' header can be used to correlate test cases with server logs for debugging.

Hint: You can visualize test results in Schemathesis.io by using `--report` in your CLI command.

========================= 1 passed, 3 failed in 0.90s ==========================

The CLI options are more or less the same and I used https://example.schemathesis.io/openapi.json as a local schema. 🤷‍♂️

Also, I don't see any changes related to max examples in 3.20, the settings are passed the same way as before. Is there any chance of e.g. custom hooks that might affect this?

Another possible option is that shrinking may affect the number of executed tests - as I don't have access to your schema, could you check with --hypothesis-phases=generate? It will disable shrinking

[dbire]

Thank you for your reply. I have tested --hypothesis-phases=generate, but it did not change the result.
The number of cases generated by schemathesis is still mysterious for me...