Backward-compatibility check for new mandatory query parameters should fail

[Denis Veljacic]

Description
When comparing two .yaml files (contracts) for backward compatibility, if the newer version has some new required query parameters, or is missing some required query params from the older version, the check should fail with the message that the newer contract is not backward compatible. Instead, it returns a successful response saying that the newer contract is backward compatible.

Steps to reproduce

1. Create a file pets-v1.yaml

openapi: 3.0.0
info:
  version: 1.0.0
  title: Test APIs
  description: APIs to test query param backward compatibility check.
tags:
  - name: Pets
servers:
  - url: 'http://someurl'
paths:
  /pets:
    get:
      summary: Get all pets
      description: Return list of all pets.
      operationId: getPets
      parameters:
        - name: type
          in: query
          required: true
          schema:
            type: string
            example: dog
          description: Type of the pet.
        - name: breed
          in: query
          schema:
            type: string
            example: breed
          description: Breed of the pet.
      tags:
        - Pets
      responses:
        '200':
          description: Successful response.
          content:
            application/json:
              schema:
                type: object
                properties: 
                  id: 
                    type: string
                    example: '12905837'
                  name: 
                    type: string
                    example: 'Garfield'
                required:
                  - id
                  - name

2. Create file pets-v2.yaml and add new mandatory query param (or remove the existing mandatory one)

openapi: 3.0.0
info:
  version: 1.0.0
  title: Test APIs
  description: APIs to test query param backward compatibility check.
tags:
  - name: Pets
servers:
  - url: 'http://someurl'
paths:
  /pets:
    get:
      summary: Get all pets
      description: Return list of all pets.
      operationId: getPets
      parameters:
        - name: age
          in: query
          required: true
          schema:
            type: string
            example: 1-10
          description: Range of the pet's age.
        - name: breed
          in: query
          schema:
            type: string
            example: breed
          description: Breed of the pet.
      tags:
        - Pets
      responses:
        '200':
          description: Successful response.
          content:
            application/json:
              schema:
                type: object
                properties: 
                  id: 
                    type: string
                    example: '12905837'
                  name: 
                    type: string
                    example: 'Garfield'
                required: 
                  - id
                  - name

3. Execute backward compatibility check: specmatic compare pets-v1.yaml pets-v2.yaml

Expected behavior
A backward compatibility check should detect the violation and output The newer contract is not backward compatible.

Current behavior
A backward compatibility check does NOT detect the violation and outputs The newer contract is backward compatible.

System Information:

• Specmatic version: 0.78.0

Additional context
The same problem exists when making mandatory keys optional and vice versa. The backward compatibility check simply doesn't take query parameters into consideration at all. One could completely remove all parameters sections from the file above or add required query parameter, the version would always be detected as backward-compatible.

[HariKrishnan]

@dveljacic Thanks for reporting the issue. Really appreciate the details. We are taking a look.

[HariKrishnan]

@dveljacic Thanks for reporting the issue. This is indeed an issue. The root cause has to do with the fact that we do not consider "required" property for query parameters. We are looking into it. If you have some time to spare, a PR would be most welcome :). cc @joelrosario @jaydeepk

[Tóth Krisztián Gyula]

I'm using Specmactic version 1.2.12. I noticed that backward compatibility check does NOT detect the violation when changing a Request body object required from true <--> false. (see spec)

Also, if there is a parse issue in the old contract, it says The newer contract is backward compatible.

[HariKrishnan]

@tkrisztian95 thanks for adding to this thread. We will take a look.

[Joel Rosario]

Hi @dveljacic, thanks for reporting this issue, and thanks as well to @tkrisztian95 for highlighting the point about requestBody with the required property set! We've added fixes for these, and they will be out soon with a new release.