Giters

jcrist / msgspec

A fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML

Home Page:https://jcristharif.com/msgspec/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Optional[Raw] results in an unexpected validation error at json.decode time

jdef opened this issue · comments

commented

Description

it seems that the json parser finds a "object" type in the JSON stream but can't assign it to a fields that expects "null" (air-gapped system, i can't copy paste).

class Wrapper(Struct):
  result: Optional[Raw]

.. and decode some JSON into it, like { "result" : { "foo": "bar" } } - and that will repro the error

commented

Thanks for opening this. Here's a complete reproducer:

import msgspec
from typing import Optional


class Wrapper(msgspec.Struct):
    result: Optional[msgspec.Raw]


msg = b'{"result": {"foo": "bar"}}'

print(msgspec.json.decode(msg, type=Wrapper))
#> Traceback (most recent call last):
#>   File "/home/jcristharif/Code/msgspec/bug.py", line 11, in <module>
#>     print(msgspec.json.decode(msg, type=Wrapper))
#> msgspec.ValidationError: Expected `null`, got `object` - at `$.result`
commented

Can you say more about your use case for the Optional[Raw] annotation? How do you plan on consuming this field post decode (or producing this field for encoding)?

The current Raw decoding logic assumed that Raw nodes won't ever be included in a union like Raw | None (same as Optional[Raw]) . What you're seeing here is a bug in this limitation actually being enforced.

That said, there's a few things we could do here:

  1. Error if Raw is ever present in a Union. The type described above would error saying you can't include Raw inside a union like Optional[Raw].
  2. Support Raw inside unions, but if Raw is ever present then we always decode it as a Raw value, even if it matches other types in the union. There's some precedence for this - this is how we handle Any today. In your case above mypy would let you pass a None or Raw to result, but when decoding you'd always end up with a Raw, even if the JSON value is null (in that case you'd get Raw(b'null'). The union support would then be most useful for cases where you might want to support passing in-memory objects when programatically constructing these, but on the decode-side always decode them as Raw.
  3. Raw is not supported in unions except for with None. So result: Raw or result: Optional[Raw] both work, but result: Raw | int would error as an unsupported type. This restriction matches what we already do for custom types. In this case null would decode as None, but any other value would decode as a Raw value.
  4. A mix of 2 and 3. In this case any union with Raw would be supported, but only Raw and None would actually be decoded, everything else would silently be ignored (like in 2).

Given these options, I'm leaning towards either 1 (easiest, best matches the main use case for Raw), or 3 (slightly harder, but may still be useful and matches a convention we had for other types). 2 also seems fine. I think option 4 would be confusing.

commented
commented

If I get a vote, it's for "Raw | None"

Sorry, 2, 3, or 4 could support that type, the question is what happens on decode.

# Given the following type definition
import msgspec

class Wrapper(msgspec.Struct):
    result: msgspec.Raw | None

# And the message
msg = b'{"result": null}'

# The decode call
msgspec.json.decode(msg, type=Wrapper)

# would result in the following values:
# 1. Would error as an unsupported type
# 2. Would decode as `Wrapper(result=Raw(b'null'))`
# 3. Would decode as `Wrapper(result=None)`
# 4. Would decode as `Wrapper(result=None)`

As an aside, your use case would probably be best supported by not using Raw | None, but rather having result default to an empty Raw() instance. The following works today:

import msgspec
from typing import Optional


class Wrapper(msgspec.Struct):
    # result is always a Raw type, will use an empty Raw if not present
    result: msgspec.Raw = msgspec.Raw()
    # you didn't say what the structure of `error` was, so guessing string
    error: str = ""


msgs = [
    b'{"result": {"x": 1}}',            # result present
    b'{"result": null}',                # result explicitly null
    b'{"error": "an error message"}',   # result missing, error present
]

for msg in msgs:
    obj = msgspec.json.decode(msg, type=Wrapper)
    # An empty `msgspec.Result()` is false-y, so you can branch on if
    # this field is Truthy to detect the presence of a result.
    if obj.result:
        print(f"result: {bytes(obj.result).decode()!r}")
    else:
        print(f"error: {obj.error!r}")

#> result: '{"x": 1}'
#> result: 'null'
#> error: 'an error message'
commented