Giters

nodatime / nodatime.serialization

Serialization projects for Noda Time

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Empty String Conversion to Nullable missing when switching from NodaTime.Serialization.JsonNet to NodaTime.Serialization.SystemTextJson

joshrap-eb opened this issue · comments

commented

Our project was previously using newtonsoft to serialize/deserialize JSON for MVC and we used the extension method provided in NodaTime.Serialization.JsonNet to configure nodatime. We recently tried switching over to NodaTime.Serialization.SystemTextJson and everything looked great at first, but then we noticed a problem.

Previously, empty strings would be converted to null for nodatime types such as LocalTime? and LocalDate? without any configuration necessary.

With the SystemTextJson extension method (ConfigureForNodaTime), it now fails for us at the model binding layer when encountering an empty string and the entire model returned to the controller is null.

Are there any plans to bring this functionality back to the SystemTextJson converters? I noticed the old converter base class does a sort of conversion for nullable types when an empty string is encountered, but the new one does not do such a conversion.

commented

Thanks for pointing this out. I'm slightly surprised at the old converter behavior, but it would make sense for it to be consistent. I'll get to this when I have a chance. I don't expect it's massively difficult.

commented

I've investigated this a bit further, and basically it looks like this is a restriction in System.Text.Json - we don't have the opportunity to use the same converter for Instant and Nullable<Instant> for example, other than in terms of the framework handling null values automatically.

It's possible that we could have separate converters for nullable and non-nullable types, but that would be fairly ugly for something which I'm not sure was a good design decision in the first place :(

commented

it looks like this is a restriction in System.Text.Json - we don't have the opportunity to use the same converter for Instant and Nullable<Instant> for example, other than in terms of the framework handling null values automatically.

@jskeet I recently ran into that restriction myself while testing one of the .NET 7 RCs in an app that uses NodaTime and System.Text.Json, so figured I'd share my solution in case it helps.

For context, I had a library type, SomeClass, with a mix of nullable and non-nullable NodaTime properties, and I wanted to add this type to my SomeJsonSerializerContext that would ship with my library when targeting net7.0. That way, my downstream consumers would be able to use the included SomeJsonSerializerContext without needing to configure their JsonSerializerOptions for NodaTime (i.e. via ConfigureForNodaTime(...)).

I was able to get things working by defining a NodaTimeJsonConverter (derived from JsonConverterFactory) that delegates to a series of NullableJsonConverterFactory<T>, each of which wraps an official converter for T (e.g. NodaConverters.{Instant,LocalDate,OffsetDateTime}Converter) and a NullableConverter<T> : JsonConverter<T?> which either handles null or delegates back to the official converter for T.

Finally, I was able to annotate the NodaTime properties of SomeType with [JsonConverter(typeof(NodaTimeJsonConverter))] and voila:

Console.WriteLine(JsonSerializer.Deserialize(
    """
    {
      "SomeInstant": "2022-11-18T00:00:00Z",
      "SomeNullableInstant": null
    }
    """,
    SomeJsonSerializerContext.Default.SomeClass));
$ dotnet run
Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to NodaTime.Instant. Path: $.SomeInstant | LineNumber: 1 | BytePositionInLine: 39.
   at System.Text.Json.ThrowHelper.ThrowJsonException_DeserializeUnableToConvertValue(Type propertyType)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Converters.JsonMetadataServicesConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryReadAsObject(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state, Object& value)
   at System.Text.Json.Serialization.Converters.LargeObjectWithParameterizedConstructorConverter`1.ReadAndCacheConstructorArgument(ReadStack& state, Utf8JsonReader& reader, JsonParameterInfo jsonParameterInfo)
   at System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo)
   at Program.<Main>$(String[] args)
$ dotnet run -p:DefineConstants=NODA_TIME_JSON_CONVERTER
SomeClass { SomeInstant = 2022-11-18T00:00:00Z, SomeNullableInstant =  }

Not necessarily saying this is the right approach for something NodaTime would ship in-the-box, but figured I'd share just in case.

SomeClass 
sealed record SomeClass(
#if NODA_TIME_JSON_CONVERTER
    [property: JsonConverter(typeof(NodaTimeJsonConverter))]
#endif
    Instant SomeInstant,
#if NODA_TIME_JSON_CONVERTER
    [property: JsonConverter(typeof(NodaTimeJsonConverter))]
#endif
    Instant? SomeNullableInstant);
SomeJsonSerializerContext 
[JsonSerializable(typeof(SomeClass))]
[JsonSourceGenerationOptions(GenerationMode = JsonSourceGenerationMode.Default)]
sealed partial class SomeJsonSerializerContext : JsonSerializerContext
{
}
NodaTimeJsonConverter 
sealed class NodaTimeJsonConverter : JsonConverterFactory
{
    static readonly JsonConverterFactory Instant = new NullableJsonConverterFactory<Instant>(NodaConverters.InstantConverter);
    static readonly JsonConverterFactory LocalDate = new NullableJsonConverterFactory<LocalDate>(NodaConverters.LocalDateConverter);
    static readonly JsonConverterFactory OffsetDateTime = new NullableJsonConverterFactory<OffsetDateTime>(NodaConverters.OffsetDateTimeConverter);

    public override bool CanConvert(Type typeToConvert)
    {
        Check.NotNull(typeToConvert);

        return GetFactory(typeToConvert) is not null;
    }

    public override JsonConverter? CreateConverter(Type typeToConvert, JsonSerializerOptions options)
    {
        Check.NotNull(typeToConvert);
        Check.NotNull(options);

        // Look for custom runtime converters from the caller first
        // TODO: https://github.com/dotnet/runtime/issues/63686
        foreach (var converter in options.Converters)
        {
            if (converter is NodaTimeJsonConverter)
                continue;

            if (!converter.CanConvert(typeToConvert))
                continue;

            return converter is JsonConverterFactory factory
                ? factory.CreateConverter(typeToConvert, options)
                : converter;
        }

        return GetFactory(typeToConvert)?.CreateConverter(typeToConvert, options);
    }

    static JsonConverterFactory? GetFactory(Type typeToConvert) =>
        Instant.CanConvert(typeToConvert) ? Instant :
        LocalDate.CanConvert(typeToConvert) ? LocalDate :
        OffsetDateTime.CanConvert(typeToConvert) ? OffsetDateTime :
        null;

    sealed class NullableJsonConverterFactory<T> : JsonConverterFactory where T : struct
    {
        readonly JsonConverter<T> _converter;
        readonly JsonConverter<T?> _nullable;

        public NullableJsonConverterFactory(JsonConverter<T> converter)
        {
            _converter = Check.NotNull(converter);
            _nullable = new NullableConverter(_converter);
        }

        public override bool CanConvert(Type typeToConvert)
        {
            Check.NotNull(typeToConvert);

            return _converter.CanConvert(typeToConvert) ||
                   _nullable.CanConvert(typeToConvert);
        }

        public override JsonConverter? CreateConverter(Type typeToConvert, JsonSerializerOptions options)
        {
            Check.NotNull(typeToConvert);
            Check.NotNull(options);

            if (_converter.CanConvert(typeToConvert))
                return _converter;

            if (_nullable.CanConvert(typeToConvert))
                return _nullable;

            return null;
        }

        sealed class NullableConverter : JsonConverter<T?>
        {
            readonly JsonConverter<T> _converter;

            public NullableConverter(JsonConverter<T> converter)
                => _converter = Check.NotNull(converter);

            public override T? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
            {
                Check.NotNull(typeToConvert);
                Check.NotNull(options);

                if (reader.TokenType is JsonTokenType.Null)
                    return default;

                return _converter.Read(ref reader, typeToConvert, options);
            }

            public override void Write(Utf8JsonWriter writer, T? value, JsonSerializerOptions options)
            {
                Check.NotNull(writer);
                Check.NotNull(options);

                if (value is null)
                {
                    writer.WriteNullValue();
                    return;
                }

                _converter.Write(writer, value.Value, options);
            }
        }
    }
}
Complete source 
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <Nullable>enable</Nullable>
    <OutputType>exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="NodaTime" Version="3.1.5" />
    <PackageReference Include="NodaTime.Serialization.SystemTextJson" Version="1.0.0" />
    <PackageReference Include="System.Text.Json" Version="7.0.0" />
  </ItemGroup>

</Project>
using System;
using System.Runtime.CompilerServices;
using System.Text.Json;
using System.Text.Json.Serialization;
using NodaTime;
using NodaTime.Serialization.SystemTextJson;

Console.WriteLine(JsonSerializer.Deserialize(
    """
    {
      "SomeInstant": "2022-11-18T00:00:00Z",
      "SomeNullableInstant": null
    }
    """,
    SomeJsonSerializerContext.Default.SomeClass));

[JsonSerializable(typeof(SomeClass))]
[JsonSourceGenerationOptions(GenerationMode = JsonSourceGenerationMode.Default)]
sealed partial class SomeJsonSerializerContext : JsonSerializerContext
{
}

sealed record SomeClass(
#if NODA_TIME_JSON_CONVERTER
    [property: JsonConverter(typeof(NodaTimeJsonConverter))]
#endif
    Instant SomeInstant,
#if NODA_TIME_JSON_CONVERTER
    [property: JsonConverter(typeof(NodaTimeJsonConverter))]
#endif
    Instant? SomeNullableInstant);

sealed class NodaTimeJsonConverter : JsonConverterFactory
{
    static readonly JsonConverterFactory Instant = new NullableJsonConverterFactory<Instant>(NodaConverters.InstantConverter);
    static readonly JsonConverterFactory LocalDate = new NullableJsonConverterFactory<LocalDate>(NodaConverters.LocalDateConverter);
    static readonly JsonConverterFactory OffsetDateTime = new NullableJsonConverterFactory<OffsetDateTime>(NodaConverters.OffsetDateTimeConverter);

    public override bool CanConvert(Type typeToConvert)
    {
        Check.NotNull(typeToConvert);

        return GetFactory(typeToConvert) is not null;
    }

    public override JsonConverter? CreateConverter(Type typeToConvert, JsonSerializerOptions options)
    {
        Check.NotNull(typeToConvert);
        Check.NotNull(options);

        // Look for custom runtime converters from the caller first
        // TODO: https://github.com/dotnet/runtime/issues/63686
        foreach (var converter in options.Converters)
        {
            if (converter is NodaTimeJsonConverter)
                continue;

            if (!converter.CanConvert(typeToConvert))
                continue;

            return converter is JsonConverterFactory factory
                ? factory.CreateConverter(typeToConvert, options)
                : converter;
        }

        return GetFactory(typeToConvert)?.CreateConverter(typeToConvert, options);
    }

    static JsonConverterFactory? GetFactory(Type typeToConvert) =>
        Instant.CanConvert(typeToConvert) ? Instant :
        LocalDate.CanConvert(typeToConvert) ? LocalDate :
        OffsetDateTime.CanConvert(typeToConvert) ? OffsetDateTime :
        null;

    sealed class NullableJsonConverterFactory<T> : JsonConverterFactory where T : struct
    {
        readonly JsonConverter<T> _converter;
        readonly JsonConverter<T?> _nullable;

        public NullableJsonConverterFactory(JsonConverter<T> converter)
        {
            _converter = Check.NotNull(converter);
            _nullable = new NullableConverter(_converter);
        }

        public override bool CanConvert(Type typeToConvert)
        {
            Check.NotNull(typeToConvert);

            return _converter.CanConvert(typeToConvert) ||
                   _nullable.CanConvert(typeToConvert);
        }

        public override JsonConverter? CreateConverter(Type typeToConvert, JsonSerializerOptions options)
        {
            Check.NotNull(typeToConvert);
            Check.NotNull(options);

            if (_converter.CanConvert(typeToConvert))
                return _converter;

            if (_nullable.CanConvert(typeToConvert))
                return _nullable;

            return null;
        }

        sealed class NullableConverter : JsonConverter<T?>
        {
            readonly JsonConverter<T> _converter;

            public NullableConverter(JsonConverter<T> converter)
                => _converter = Check.NotNull(converter);

            public override T? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
            {
                Check.NotNull(typeToConvert);
                Check.NotNull(options);

                if (reader.TokenType is JsonTokenType.Null)
                    return default;

                return _converter.Read(ref reader, typeToConvert, options);
            }

            public override void Write(Utf8JsonWriter writer, T? value, JsonSerializerOptions options)
            {
                Check.NotNull(writer);
                Check.NotNull(options);

                if (value is null)
                {
                    writer.WriteNullValue();
                    return;
                }

                _converter.Write(writer, value.Value, options);
            }
        }
    }
}

static class Check
{
    public static T NotNull<T>(T? value, [CallerArgumentExpression("value")] string name = null!) => value ?? throw new ArgumentNullException(name);
}
commented

@austindrenski: Good to know as a workaround, thanks. Given how much it requires going against the framework, I don't want that to be the default behavior. I might look at how users could opt into it though...

commented

@jskeet
Had a similar problem with conversion from empty (not null) to LocalDateTime? und could solve it for me with a wrapper around LocalDateTimePattern.

Main File for the parsing

Important parts

  • Format: return String.empty if null
    • that way I can store back string.empty when serializing to json again, otherwise I would store null in the field
    • the json-format itself is not in my control, so I need to store back empty-string
    • you'll also need NodaNullablePatternConverter (see below) for this
    • if it doesn't matter storing back "null" you don't need to handle this here any special
  • Parse: handles empty values and returns null als result
using System.Text;
using NodaTime;
using NodaTime.Text;

namespace DateTest;

public class LocalDateTimeNullablePattern : IPattern<LocalDateTime?>
{
    private LocalDateTimePattern Pattern { get; }

    internal static IPattern<LocalDateTime?> CreateWithInvariantCulture(string pattern)
    {
        return new LocalDateTimeNullablePattern(
            LocalDateTimePattern.CreateWithInvariantCulture(pattern)
        );
    }

    private LocalDateTimeNullablePattern(LocalDateTimePattern pattern)
    {
        Pattern = pattern;
    }

    public StringBuilder AppendFormat(LocalDateTime? value, StringBuilder builder)
    {
        if (!value.HasValue)
            return new StringBuilder();

        return Pattern.AppendFormat(value.Value, builder);
    }

    public string Format(LocalDateTime? value)
    {
        if (!value.HasValue)
            return string.Empty;
        return Pattern.Format(value.Value);
    }

    public ParseResult<LocalDateTime?> Parse(string text)
    {
        var parseResult = Pattern.Parse(text);
        if (!parseResult.Success)
        {
            if (string.IsNullOrEmpty(text))
                return ParseResult<LocalDateTime?>.ForValue(null);

            return Pattern.Parse(text).ConvertError<LocalDateTime?>();
        }
        return Pattern.Parse(text).Convert(s => (LocalDateTime?)s);
    }
}
Nullable Converter

Important parts

  • see description above for "Format". This file is only needed if you want to write back an empty string (instead null) on Serialize
  • important is the HandleNull => true, so that SystemTextJson doesn't handle null values on its own, but let the Converter decide how to serialize this
  • it may also be an option to extend the original NodaPatternConverter with "HandleNull => true" and hope that nothing breaks.
using NodaTime.Serialization.SystemTextJson;

namespace DateTest;

public class NodaNullablePatternConverter<T> : DelegatingConverterBase<T>
{
    public override bool HandleNull => true;

    public NodaNullablePatternConverter(NodaTime.Text.IPattern<T> pattern)
        : base(new NodaPatternConverter<T>(pattern)) { }
}
My (simple) Model 
using NodaTime;

internal class Model
{
    public LocalDateTime? Test { get; set; }
}
My Testprogram

Information

  • there are 2 sections:
    • the first converting from empty string to nullable LocalDateTime and back
    • the second converting a "normal" date string to nullable LocalDateTime and back
using System.Text.Json;
using DateTest;
using NodaTime;

var pattern = LocalDateTimeNullablePattern.CreateWithInvariantCulture("yyyy-MM-dd HH:mm:ss.fff");

var options = new JsonSerializerOptions
{
    Converters = { new NodaNullablePatternConverter<LocalDateTime?>(pattern) }
};

// Converting empty string
var jsonString = "{\"Test\": \"\" }";
var model = JsonSerializer.Deserialize<Model>(jsonString, options);
if (model != null)
{
    var serializedString = JsonSerializer.Serialize<Model>(model, options);
    Console.WriteLine($"original: {jsonString}");
    Console.WriteLine($"serialized: {serializedString}");
}

// Converting valid datestring for pattern
jsonString = "{\"Test\": \"2018-07-20 09:16:35.617\" }";
model = JsonSerializer.Deserialize<Model>(jsonString, options);
if (model != null)
{
    var serializedString = JsonSerializer.Serialize<Model>(model, options);
    Console.WriteLine($"original: {jsonString}");
    Console.WriteLine($"serialized: {serializedString}");
}
Output image
commented

Even better:

  • Using LocalDateTimeNullablePattern<T> instead of LocalDateTimeNullablePattern
  • T can take a LocalDateTimePattern or LocalDateTimePattern? (ctor checks for the correct type and fails at run time if not true - that may be a point to discuss)
  • advantage: I can use these patterns without creating additional classes:
var nullablePattern = LocalDateTimeNullablePattern<LocalDateTime?>.CreateWithInvariantCulture(
    "yyyy-MM-dd HH:mm:ss.fff"
);
var nonNullablePattern = LocalDateTimeNullablePattern<LocalDateTime>.CreateWithInvariantCulture(
    "yyyy-MM-dd HH:mm:ss.fff"
);
  • Usage in converter
Converters =
{
      new NodaNullablePatternConverter<LocalDateTime?>(nullablePattern),
      new NodaNullablePatternConverter<LocalDateTime>(nonNullablePattern)
}
Main File for the parsing 
using System.Text;
using NodaTime;
using NodaTime.Text;

namespace DateTest;

public class LocalDateTimeNullablePattern<T> : IPattern<T>
{
    private LocalDateTimePattern Pattern { get; }

    internal static IPattern<T> CreateWithInvariantCulture(string pattern)
    {
        return new LocalDateTimeNullablePattern<T>(
            LocalDateTimePattern.CreateWithInvariantCulture(pattern)
        );
    }

    private LocalDateTimeNullablePattern(LocalDateTimePattern pattern)
    {
        if (BaseType != typeof(LocalDateTime))
            throw new ArgumentException(
                $"Type {typeof(T)} is not allowed here, only LocalDateTime|Nullable<LocalDateTime> allowed."
            );
        Pattern = pattern;
    }

    public StringBuilder AppendFormat(T value, StringBuilder builder)
    {
        if (!HasValue(value))
            return new StringBuilder();

        return Pattern.AppendFormat(GetLocalDateTime(value), builder);
    }

    public string Format(T value)
    {
        if (!HasValue(value))
            return string.Empty;
        return Pattern.Format(GetLocalDateTime(value));
    }

    public ParseResult<T> Parse(string text)
    {
        var parseResult = Pattern.Parse(text);

        if (!parseResult.Success)
        {
#pragma warning disable CS8604
            //Checking for IsValueType wouldn't be necessary, since LocalDateTime is always valuetype, compiler is still complaining that ForValue may get a null value
            if (string.IsNullOrEmpty(text) && IsNullable && typeof(T).IsValueType)
            {
                return ParseResult<T>.ForValue(default);
            }
#pragma warning restore CS8604

            return Pattern.Parse(text).ConvertError<T>();
        }
        return parseResult.Convert(GetT);
    }

    private readonly Type BaseType = Nullable.GetUnderlyingType(typeof(T)) ?? typeof(T);

    private readonly bool IsNullable = Nullable.GetUnderlyingType(typeof(T)) != null;

    private static bool HasValue(T value) => !EqualityComparer<T>.Default.Equals(value, default);

    private static LocalDateTime GetLocalDateTime(T value) =>
        value == null ? default : (LocalDateTime)Convert.ChangeType(value, typeof(LocalDateTime));

    private T GetT(LocalDateTime value) => (T)Convert.ChangeType(value, BaseType);
}
commented

@Cyber1000: If you'd like to create a PR for this, that would be very welcome. #88 shows how I'm planning to make the converters easier to configure. (That's just for Json.NET, but I'd do something equivalent for SystemTextJson.)

commented

thanks, I'll have a look into this (disclaimer: low on time, may take some time)