Incorrect handling of marshmallow nested schema with allow_none=True in OpenAPI 3.1

[James Harrison]

If I define a marshmallow schema like this:

from marshmallow import fields, Schema

class FooSchema(Schema):
    bar = fields.Integer(required=True)

class MySchema(Schema):
    foo = fields.Nested(FooSchema, allow_none=True)

it behaves as I intend - MySchema can load objects with a foo field that either matches FooSchema or is null, but FooSchema itself doesn't allow null values:

>>> MySchema().loads('{"foo": {"bar": 1}}')
{'foo': {'bar': 1}}
>>> MySchema().loads('{"foo": null}')
{'foo': None}
>>> FooSchema().loads('null')
marshmallow.exceptions.ValidationError: {'_schema': ['Invalid input type.']}

But if I generate an API spec with this schema, using openapi version 3.1:

from apispec import APISpec
from apispec.ext.marshmallow import MarshmallowPlugin

spec = APISpec(
    title="My API Spec",
    version="1.0.0",
    openapi_version="3.1.0",
    plugins=[MarshmallowPlugin()],
)
spec.components.schema("MySchema", schema=MySchema)

the resulting API spec is not equivalent to my marshmallow schema:

>>> print(json.dumps(spec.to_dict()["components"]["schemas"], indent=2))
{
  "Foo": {
    "type": "object",
    "properties": {
      "bar": {
        "type": "integer"
      }
    },
    "required": [
      "bar"
    ]
  },
  "MySchema": {
    "type": "object",
    "properties": {
      "foo": {
        "type": [
          "null"
        ],
        "allOf": [
          {
            "$ref": "#/components/schemas/Foo"
          }
        ]
      }
    }
  }
}

This spec says that the value of foo must be null, AND must be a Foo object, so in fact there is no valid input value here.

I think a correct spec in this instance should contain:

{
  "Foo": {
    "type": "object",
    "properties": {
      "bar": {
        "type": "integer"
      }
    },
    "required": [
      "bar"
    ]
  },
  "MySchema": {
    "type": "object",
    "properties": {
      "foo": {
        "anyOf": [
          {
            "$ref": "#/components/schemas/Foo"
          },
          {
            "type": "null"
          }
        ]
      }
    }
  }
}

I'm using apispec 6.3.0 and marshmallow 3.19.0

[Jérôme Lafréchoux]

I'm afraid you're right. Looks like a bug.

Would you like to investigate this and propose a fix?

[James Harrison]

Would you like to investigate this and propose a fix?

I'm not sure how soon I'll have time to investigate this further, I'm afraid, but I'll try to look into it at some point.

[Michael Bangert]

@lafrech the problem here seems to be that field2nullable and nested2properties are operating independently. It looks like for OpenAPI 3.1 they are going to need to interoperate.

For this schema under 3.1 field2nullable is adding "type": ["null"] and nested2properties is independently adding "allOf": [{"$ref": "#/components/schemas/Foo"}]

This approach was probably fine pre 3.1, but now that nullability needs to be included in an anyOf array we need make one method depend on the result of the other.

This is probably also an issue for pluck2properties, but there the symptom would be not displaying nullability because the type is always being overwritten.

The solution I'm thinking about is:

1. Moving the execution of field2nullable to be the last attribute function executed. I think this change alone solves the Pluck case and the case of nested schemas defined inline.
2. Adding logic under the 3.1 case to translate the allOf keyword to anyOf while adding the null type.
3. We may also want to ensure that field2nullable executes after user defined attribute functions to prevent similar issues with those.