System.Text.Json and Custom Formats in Model Binding
apavelm opened this issue · comments
Hi there!
I've spent the whole day trying to solve or bypass the issue I described below, any help or confirmation about existing bug will be very appreciated.
Brief description:
Previously, I figured out (thank you for pointing me to the right way, again) how to add custom converter/formats for further deserialization. So, I have the following configuration:
public static NodaJsonSettings GetNodaJsonSettings()
{
var localTimePatternBuilder = new CompositePatternBuilder<LocalTime>();
localTimePatternBuilder.Add(LocalTimePattern.CreateWithInvariantCulture("HH:mm"), time => true);
localTimePatternBuilder.Add(LocalTimePattern.ExtendedIso, time => true);
var localTimePattern = localTimePatternBuilder.Build();
var localDateTimePatternBuilder = new CompositePatternBuilder<LocalDateTime>();
localDateTimePatternBuilder.Add(LocalDateTimePattern.CreateWithInvariantCulture("yyyy-MM-dd HH:mm"), time => true);
localDateTimePatternBuilder.Add(LocalDateTimePattern.CreateWithInvariantCulture("yyyy-MM-ddTHH:mm"), time => true);
localDateTimePatternBuilder.Add(LocalDateTimePattern.ExtendedIso, time => true);
var localDateTimePattern = localDateTimePatternBuilder.Build();
var instantPatternBuilder = new CompositePatternBuilder<Instant>();
instantPatternBuilder.Add(InstantPattern.CreateWithInvariantCulture("yyyy-MM-dd HH:mm"), time => true);
instantPatternBuilder.Add(InstantPattern.CreateWithInvariantCulture("yyyy-MM-ddTHH:mm"), time => true);
instantPatternBuilder.Add(InstantPattern.ExtendedIso, instant => true);
var instantPattern = instantPatternBuilder.Build();
var nodaJsonSettings = new NodaJsonSettings(DateTimeZoneProviders.Tzdb)
{
InstantConverter = new NodaTime.Serialization.SystemTextJson.NodaPatternConverter<Instant>(instantPattern),
LocalTimeConverter = new NodaTime.Serialization.SystemTextJson.NodaPatternConverter<LocalTime>(localTimePattern),
LocalDateTimeConverter = new NodaTime.Serialization.SystemTextJson.NodaPatternConverter<LocalDateTime>(localDateTimePattern),
};
return nodaJsonSettings;
}
And everything worked fine with Newtonsoft.JsonNET, but now we started a migration to System.Text.Json
Default converters are working fine, but not custom ones.
I tried many ways on how to fix the issue, but it seems like on Model Binding all custom formats are ignored despite it exists in configuration.
I've read several similar issue reports and did as many bypassed as possible to push the only one correct JsonSettingsOptions configuration:
var jsonSerializerOptions= new JsonSerializerOptions()
{
PropertyNameCaseInsensitive = true,
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
DictionaryKeyPolicy = JsonNamingPolicy.CamelCase,
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
ReferenceHandler = ReferenceHandler.IgnoreCycles,
WriteIndented = true
};
jsonSerializerOptions.Converters.Add(new JsonStringEnumConverter());
jsonSerializerOptions.ConfigureForNodaTime(SerializationExtension.GetNodaJsonSettings());
services.AddSingleton(s => jsonSerializerOptions);
services.AddControllers(
config =>
{
config.Filters.Add<SpotoExceptionFilter>();
var inputFormatter =
config.InputFormatters.OfType<SystemTextJsonInputFormatter>().FirstOrDefault();
if (inputFormatter != null)
{
SerializationExtension.CopyJsonSerializerOptions(inputFormatter.SerializerOptions, jsonSerializerOptions);
}
var outputFormatter =
config.OutputFormatters.OfType<SystemTextJsonInputFormatter>().FirstOrDefault();
if (outputFormatter != null)
{
SerializationExtension.CopyJsonSerializerOptions(outputFormatter.SerializerOptions, jsonSerializerOptions);
}
})
.AddJsonOptions(o => SerializationExtension.CopyJsonSerializerOptions(o.JsonSerializerOptions, jsonSerializerOptions));
Where there is a helper method:
public static void CopyJsonSerializerOptions(JsonSerializerOptions targetOptions, JsonSerializerOptions sourceOptions)
{
// Notes. A hacky but necessary approach, because we are given a pre-built JsonSerializerOptions object
// that can't be replaced, therefore we must copy the various setting into it.
targetOptions.AllowTrailingCommas = sourceOptions.AllowTrailingCommas;
targetOptions.DefaultBufferSize = sourceOptions.DefaultBufferSize;
targetOptions.DefaultIgnoreCondition = sourceOptions.DefaultIgnoreCondition;
targetOptions.DictionaryKeyPolicy = sourceOptions.DictionaryKeyPolicy;
targetOptions.Encoder = sourceOptions.Encoder;
targetOptions.IgnoreReadOnlyFields = sourceOptions.IgnoreReadOnlyFields;
targetOptions.IncludeFields = sourceOptions.IncludeFields;
targetOptions.MaxDepth = sourceOptions.MaxDepth;
targetOptions.NumberHandling = sourceOptions.NumberHandling;
targetOptions.PropertyNameCaseInsensitive = sourceOptions.PropertyNameCaseInsensitive;
targetOptions.PropertyNamingPolicy = sourceOptions.PropertyNamingPolicy;
targetOptions.ReadCommentHandling = sourceOptions.ReadCommentHandling;
targetOptions.ReferenceHandler = sourceOptions.ReferenceHandler;
targetOptions.UnknownTypeHandling = sourceOptions.UnknownTypeHandling;
targetOptions.WriteIndented = sourceOptions.WriteIndented;
targetOptions.DefaultIgnoreCondition = sourceOptions.DefaultIgnoreCondition;
// Re-use the same converter instances; this is OK because their current parent
// SystemTextJsonInputFormatter is about to be discarded.
targetOptions.Converters.Clear();
foreach (var jsonConverter in sourceOptions.Converters)
{
targetOptions.Converters.Add(jsonConverter);
}
}
But nothing helped.
And when I send a request to a test endpoint:
public class TestReq
{
public LocalDateTime datetime { get; set; } // 2023-11-01 12:15
public LocalTime time { get; set; } // 13:45
}
public IActionResult Test([FromQuery] TestReq req)
I'm getting and wrong model exception, but when used Newtonsoft.JsonNet it worked like a charm.
@jskeet Is it a known bug or there is a way how bypass it? Any info will be very welcomed.
I'm afraid I know nothing about model binding or how it varies based on JSON converter.
I'm unlikely to be able to look into this for at least a couple of weeks - quite possibly not until Christmas I'm afraid.
What I'd start with is trying to get it working with the default patterns instead of custom ones, so we can distinguish between "It's not picking up on Noda Time configuration at all" and "It's not picking up on custom configuration".
@jskeet Default converters are working fine; otherwise there will be a dozen of bug-reports, haha.
The issue is only with custom ones. And It's difficult to identify on what level them are ignored.
Thanks
Added
Maybe it is more related to dotnet/runtime#31094 but I cannot prove it now.
Can you try your custom converters for situations other than model binding?
How about "simple" custom converters that don't use CompositePattern?
Can you try your custom converters for situations other than model binding? How about "simple" custom converters that don't use CompositePattern?
The same. tried for NodaPatternConverter<LocalTime>(LocalTimePattern.CreateWithInvariantCulture("HH:mm"))
The most weird thing, it works fine for default converters/formats but not for custom ones. Because of that, it is not clear where the issue is.
Oh, if custom converters aren't working at all (separately from model binding) that suggests a bug in the fairly recent custom converter code.
If sounds like you should be able to demonstrate that in a standalone console app, which would make it easier to test...
(Basically, if we can get to that stage, I'd anticipate being able to fix it and release it quite quickly. But I won't have time to get up and running with model binding etc.)
@jskeet ,
I did some tests and must conclude that it appears not a nodatime issue.
The root-cause of the problems is inability to override default deserializer configuration - the best option is to have static member for that and specify it explicitly every time (I mean System.Text.Json)
Accidentally, I missed the test case, and now I confirm that it is working: POST requests deserializing well.
The problem with deserialization on GET requests (FromQuery attribute) happens for both: STJ and JsonNET - hence this is not NodaTime problem.
But I would appreciate it if we could add some converters using default methods to ModelBinding.
Thank you for your readiness to help!
On GET requests when the request is not JSON,, but like www-form-data or just query parameters.
NodaTime.Serialization not used, it uses TypeConverters defined here https://github.com/nodatime/nodatime/blob/ef7c92f2bb10ce598358ff0b3e94e6d3d6d9f51f/src/NodaTime/Text/TypeConverters.cs#L34 All of them doesn't consider customer and complex patterns and uses only default ones. That is why default formats were deserialzed OK, but any custom didn't
@jskeet I'm not sure if it should be considered as a bug or not and maybe I should create a ticket under nodatime project, but how can we add custom converter formats there?
Thanks
nodatimeserialization.zip
just a test project
Given the way type converters work, I don't think there's a clean way of making them configurable, so I don't think it's worth creating an issue for that I'm afraid.
@jskeet It appears like we don't need custom Converters. All what we need is to slightly modify TypeBaseConverter class.
I prepared a small modification to NodaTime library that does this job.
I'm not very familiar with NodaTime internal design so, please consider this as a draft. Maybe you would like to move/rename or do the same in a different way.
// Copyright 2018 The Noda Time Authors. All rights reserved.
// Use of this source code is governed by the Apache License 2.0,
// as found in the LICENSE.txt file.
using JetBrains.Annotations;
using NodaTime.Utility;
using System;
using System.ComponentModel;
using System.Globalization;
using NodaTime.Extensions;
namespace NodaTime.Text
{
/// <summary>
/// Provides conversion and parsing for <typeparamref name="T"/>.
/// </summary>
internal abstract class TypeConverterBase<T> : TypeConverter
{
/// <summary>
/// The pattern used to parse and serialize values of <typeparamref name="T"/>.
/// </summary>
private readonly IPattern<T> pattern;
/// <summary>
/// Constructs a <see cref="TypeConverter"/> for <typeparamref name="T"/> based on the provided <see cref="IPattern{T}"/>.
/// </summary>
/// <param name="defaultPattern">The pattern used to parse and serialize <typeparamref name="T"/>.</param>
/// <exception cref="ArgumentNullException"><paramref name="defaultPattern"/></exception>
protected TypeConverterBase(IPattern<T> defaultPattern) => this.pattern = Preconditions.CheckNotNull(defaultPattern, nameof(defaultPattern));
protected IPattern<T> GetPattern()
{
var patterns = NodaTimeCustomPatternExtensions.GetCustomPatterns<T>();
if (patterns is null)
return pattern;
else
{
var builder = new CompositePatternBuilder<T> {{pattern, instant => true}};
foreach (var p in patterns)
{
builder.Add(p, instant => true);
}
var compositePattern = builder.Build();
return compositePattern;
}
}
/// <inheritdoc />
[Pure]
public override bool CanConvertFrom(ITypeDescriptorContext? context, Type sourceType) => sourceType == typeof(string);
/// <inheritdoc />
[Pure]
public override object? ConvertFrom(ITypeDescriptorContext? context, CultureInfo? culture, object value) =>
// The ParseResult<>.Value property will throw appropriately if the operation was unsuccessful
value is string text ? GetPattern().Parse(text).Value! : base.ConvertFrom(context, culture, value);
/// <inheritdoc />
[Pure]
public override object? ConvertTo(ITypeDescriptorContext? context, CultureInfo? culture, object? value, Type destinationType) =>
destinationType == typeof(string) && value is T nodaValue
? GetPattern().Format(nodaValue)
: base.ConvertTo(context, culture, value, destinationType);
}
}
And a new extensions helper class:
using System;
using System.Collections.Generic;
using NodaTime.Text;
using NodaTime.Utility;
namespace NodaTime.Extensions
{
/// <summary>
/// Extension that allow manage <see cref="IPattern{T}"/>. in a dynamic custom way for default converters
/// </summary>
public static class NodaTimeCustomPatternExtensions
{
internal static Dictionary<Type, object> CustomPatternTable = new Dictionary<Type, object>();
internal static IPattern<T>[]? GetCustomPatterns<T>()
{
if (CustomPatternTable.ContainsKey(typeof(T)))
{
var list = CustomPatternTable[typeof(T)] as List<IPattern<T>>;
return list?.ToArray();
}
return null;
}
/// <summary>
/// Add new IPattern for a NodaTime type to custom pattern dictionary
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="pattern"></param>
public static void AddNodaTimeCustomTypePattern<T>(IPattern<T> pattern)
{
Preconditions.CheckNotNull(pattern, nameof(pattern));
List<IPattern<T>> list = new List<IPattern<T>>();
if (CustomPatternTable.ContainsKey(typeof(T)))
{
list = (CustomPatternTable[typeof(T)] as List<IPattern<T>>)!;
}
list.Add(pattern);
CustomPatternTable[typeof(T)] = list;
}
}
}
Please let me know if you want me to prepare PR.
It doesn't modify the existing behavior and allow to use correct custom pattern on deserialization by default.
@jskeet Looking forward to hearing from you. We need this or similar extension/fix for nodatime because using nodatime as sources in own project is not a perfect idea.
I'm afraid I'm not going to have time to look at this in any detail any time soon - but yes, please create a pull request as it'll be much easier to compare it that way. It's worth noting that minor versions of Noda Time are significantly more work than for the serialization libraries, so I try to do them having batched up changes. I may end up doing a release over Christmas/New Year, but it's very unlikely to be before that.
Off-hand, this looks like mutable global data - accessed in a non-thread-safe way, so I'm really not a big fan at a glance.
That is like a PoC that demonstrates the idea. As I wrote, I'm not familiar with the internal design of NodaTime, so I assume you'd like to refactor it anyway.
BTW, here is a PR: nodatime/nodatime#1766
P.S. You mentioned the X-Mas update release. Will it be based on a new .NET 8 LTS, right?
P.S. You mentioned the X-Mas update release. Will it be based on a new .NET 8 LTS, right?
No, I don't plan to target .NET 8 yet. I'll only do so when I want to take advantage of new APIs - ITimeProvider in particular. I'm not expecting to find enough time to do that soon.
Noda Time will still work on .NET 8 of course, using the net6.0 target.
Hm, probably someone should add a new target by 2 reasons:
- .NET 8 is the next LTS release
- Performance improvements in .NET 8. Most likely, some internal improvements from NET 8 will affect NodaTime as well. Curious to see performance benchmark.
Alright, @jskeet thanks for answers, please notify me somehow once a modification will be merged/release. As I wrote it is not a good idea to keep nodatime sources inside the solution :-)
All best!
Neither of those are reasons for adding a new target. We could easily run benchmarks on .NET 8 without changing the targets of Noda Time. (And we should see any performance improvements in .NET 8 that way.)
As it is, the code compiled for a .NET 8 target would be identical to the code compiled for a .NET 6 target... and when consuming Noda Time from a .NET 8 app, it'll just use the .NET 6 target. So there's really no benefit from adding a .NET 8 target at the moment - it would just make the nuget package bigger.