HTTP POST - InvalidNodaDataException for null values (NewtonsoftJson)

[Jaromír Kejda]

Hello,

we have been using the NodaTime library for building ASP.NET Core REST APIs and we would like to handle incoming null values from the client more appropriately. We have observed, that null value for types such as LocalDateTime, Instant (and possibly others) results in an exception being thrown and HTTP 500 status code, for example: NodaTime.Utility.InvalidNodaDataException: Cannot convert null value to NodaTime.LocalDateTime.

On the other hand, when assigning a null value to the BCL DateTime property, HTTP 400 status code is returned with an error message Error converting value {null} to type 'System.DateTime'. From our point of view, this behaviour is more appropriate, because it is clearly the fault of the client, so the HTTP 400 makes more sense.

The following overview compares the difference in behaviour between the LocalDateTime and DateTime types (using HTTP POST request with enabled ConfigureForNodaTime(DateTimeZoneProviders.Tzdb))

The HTTP POST examples

{
    // *** NODA ***
     
    // 1) HTTP 200
    // localDateTime": "2022-01-01T10:00:00",
   
    // 2) HTTP 500, NodaTime.Utility.InvalidNodaDataException: Cannot convert null value to NodaTime.LocalDateTime
    // "localDateTime": null,

    // 3) HTTP 400, Cannot convert value to NodaTime.LocalDateTime
    // "localDateTime": false

    // 4) HTTP 400, Cannot convert value to NodaTime.LocalDateTime
    // "localDateTime": "abcd"

    // *** BCL ***

    // 5) HTTP 200
    // "dateTime": "2022-01-01T00:00:00Z"
   
    // 6) HTTP 400,   Error converting value {null} to type 'System.DateTime'.
    // "dateTime": null

    // 7) HTTP 400, Unexpected character encountered while parsing value: f
    // "dateTime": false

    // 8) HTTP 400, Could not convert string to DateTime: abcd.
    // "dateTime": "abcd"
}

Program.cs

using NodaTime;
using NodaTime.Serialization.JsonNet;

var builder = WebApplication.CreateBuilder(args);

// services
builder.Services.AddControllers().AddNewtonsoftJson(s =>
    s.SerializerSettings.ConfigureForNodaTime(DateTimeZoneProviders.Tzdb));

builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
var app = builder.Build();

// pipeline
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}
app.UseAuthorization();
app.MapControllers();
app.Run();

Endpoint and model

namespace NodaNewtonsoft.Controllers;

[ApiController]
[Route("demo")]
public class DemoController : ControllerBase
{

    [HttpPost("noda")]
    public IActionResult Demo([FromBody] Request request)
    {
        return Ok(request);
    }
}

public class Request
{
    public LocalDateTime LocalDateTime { get; set; }
    public DateTime DateTime { get; set; }
}

The bottom line is, we do not want to return HTTP 500 for scenarios when the provided value for the LocalDateTime is null. Although making the property nullable fixes the issue, it does not seem like the right solution for the cases when the value must be provided by the client. Is this behaviour by design? Is there any recommended way to get the standard 400 Bad Request result for the null values, such as:

{
    "errors": {      
        "localDateTime": [
            // some example error message that does not exist right now
            "Cannot convert {null} to NodaTime.LocalDateTime"
        ]
    },
    "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
    "title": "One or more validation errors occurred.",
    "status": 400,
    "traceId": "00-38367af21814a198e948f57c45737200-077bfffefaea8feb-00"
}

Thank you for your help and have a nice day.

Details

• "Microsoft.AspNetCore.Mvc.NewtonsoftJson", Version="6.0.11"
• "NodaTime", Version="3.1.5"
• "NodaTime.Serialization.JsonNet", Version="3.0.1"
• Target framework - net6.0

The runnable demo app is attached below.
NodaNewtonsoft.zip

[Jon Skeet]

(After posting this comment, I'll move this to the serialization repository which is where all the serialization code lives.)

Are you able to reproduce this without using Swagger and ASP.NET Core? If you could provide a simple JSON document and show the different exceptions when deserializing to LocalDate vs DateTime, that would be very helpful.

Is this behaviour by design?

We don't have any behavior which is specific to Swagger or to ASP.NET Core. We just have preconditions which throws exceptions when they're violated, and presumably the framework above this is converting that into an HTTP status code.

The interesting points here, I suspect, are the precondition call, the CheckData method, and InvalidNodaDataException which is thrown (and which just extends Exception).

You could potentially argue that it should throw ArgumentException instead, but it's not that the argument itself (a JsonReader) is invalid, but that the data within it is invalid. Either way, it would be a seriously breaking change to alter which exception is thrown now.

I suspect your best way forward is to add something to the pipeline to convert InvalidNodaDataException into a 400 response instead of a 500 response - if that's the only place you can ever get that exception, of course.

[Jaromír Kejda]

Hello,

here is the isolated problem (.NET Console App) that shows that different exceptions are thrown:

/* installed packages
   <PackageReference Include="NodaTime" Version="3.1.5" />
   <PackageReference Include="NodaTime.Serialization.JsonNet" Version="3.0.1" /> */

// usings
using Newtonsoft.Json;
using NodaTime;
using NodaTime.Serialization.JsonNet;

// settings
var settings = new JsonSerializerSettings().ConfigureForNodaTime(DateTimeZoneProviders.Tzdb);

// methods
void BclAndNoda()
{
    // deserialization with values --> OK
    var jsonRequest = @"{""LocalDateTime"":""2022-01-01T00:00:00"",""DateTime"":""2022-01-01T00:00:00+01:00""}";
    var deserializedRequest = JsonConvert.DeserializeObject<Request>(jsonRequest, settings);
}
void Bcl()
{
    // deserialization without value --> Newtonsoft.Json.JsonSerializationException: 'Error converting value {null} to type 'System.DateTime'.
    var jsonRequestWithNull = @"{""LocalDateTime"":""2022-01-01T00:00:00"",""DateTime"":null}";
    var deserializedRequestWithNull = JsonConvert.DeserializeObject<Request>(jsonRequestWithNull, settings);

}
void Noda()
{
    // deserialization without value --> NodaTime.Utility.InvalidNodaDataException: 'Cannot convert null value to NodaTime.LocalDateTime'
    var jsonRequestWithNull = @"{""LocalDateTime"":null,""DateTime"":""2022-01-01T00:00:00+01:00""}";
    var deserializedRequestWithNull = JsonConvert.DeserializeObject<Request>(jsonRequestWithNull, settings);
}

// deserialize with non-null values --> works
BclAndNoda();

// null DateTime throws JsonSerializationException
Bcl();

// null LocalDateTime throws InvalidNodaDataException
Noda();

Console.ReadKey();

// definitions
public class Request
{
    public LocalDateTime LocalDateTime { get; set; }
    public DateTime DateTime { get; set; }
}

Unfortunately, the ASP.NET Core framework handles these exceptions differently.
Thank you for your assistance, the runnable console app is attached below.
NodaNewtonsoftConsole.zip

[Jon Skeet]

Right, thanks for the standalone example. As noted before, changing the exception we throw would be a breaking change.
I would expect that you could configure your server to handle InvalidNodaDataException in whatever way you want, however.

[pikausp]

@jskeet was it a design choice to not throw a serialization exception when attempting to deserialize a null value? One is thrown when you attempt to parse 2012 into Instant for example.

Sadly ASP.NET seems to not care about other exceptions when it's executing model binding. Sure it is possible to handle the InvalidNodaDataException within the request pipeline, but I see the two following challenges

• You lose valuable information such as what property was just being parsed that caused this error etc.
• If you handle the exception within the pipeline it would catch the same type of exception being thrown from an action code long after the model binding was done and it would be hard to differentiate between the two. It would be possible to use a marker that model binding was completed and differentiate logic based on that, but it seems like somewhat of a hack

Is it possible you'd consider making the thrown exception always a serialization exception no matter the type of issue? (null value, format,..) I do understand it is a breaking change, but at the same time, I would say that the current behavior doesn't seem to be consistent and in line with Newtonsoft's approach.

[Jon Skeet]

@pikausp: I don't remember the precise reasoning for decisions made 10 years ago, I'm afraid.

I really don't want to make a breaking change in the current major version. We could potentially change it for the next major version of this package, but I'm not planning on doing that any time soon.
Another alternative would be to allow converters to be created with options - we could keep the default option, but have another "always throw serialization exceptions" option that you'd need to opt into. We'd need to provide a way of supplying options in ConfigureForNodaTime. I'd be happy to look into that, but I wouldn't expect to do it soon.