Encode.undefined for omitting a field produced by Encode.object?

[Witold Szczerba]

I think it would be useful to be able to define encoder like this:

type SomeXY = X of string | Y
let decodeSomeXY = 
    function
    | X s -> Encode.string s
    | Y   -> Encode.undefined // or some better name?

Now, it would work like this:

> Encode.object
      [ "a", Decode.int 1
        "b", decodeSomeXY (X "…")
        "c", decodeSomeXY Y ] // this field would not be included in JSON
 |> Encode.toString 0
- ;;
val it : string = "{"a":1, "b":"…"}"

That would make it easy to create decoders with optional fields without having to (sometimes deep) filter the array of (key, values). It would also be similar to JavaScript's JSON.stringify:

> JSON.stringify({ a: 1, b: "…", c: undefined })
'{"a":1,"b":"…"}'

What would you say?

[Maxime Mangel]

Hello,

to me, it feels really strange to not have a valid JSON representation for each case of a DUs. What is your use case for having a type partially represented in term of JSON? I am just trying to understand the whole picture as JSON representation is not something easy.

Currently, in Thoth.Json we don't have a case where we don't represent one part of a DUs. The closest thing is the implementation of the Auto module for the Record.

We decided to treat in the same way the absence of a property and the value null for it. It is possible to skipNullField in order to not generates them in the output.

If you want to omit a field from an object, you can always not yield his value.

Encode.object [
    "name", Encode.string "Maxime"
    if condition then
        "x", Encode.undefined
]

But it is true that it is the "object" which decide to not emit the value, not the underlying type.

Thoth.Json is not tied to a specific JSON parser, so we can't rely on a parser implementation to do the job for us:

• Thoth.Json use native JavaScript parser
• Thoth.Json.Net use Newtonsoft.Net
• Thoth.Json vNext use a custom parser

A possible workflow for the "implementation" could be:

type ErasedType =
    | String of string
    | Nothing

    static member Encoder (v : ErasedType)=
        match v with
        | String str -> Encode.string str
        | Nothing -> Encode.erased

Encode.object [
    "name", Encode.string "Maxime"
    "x", ErasedType.Encoder Nothing
] // Encode.object could apply a filter to remove the value makes with "erased" 
// Implementation detail: Because we are already iterating over the list once to create the object, we should just add a check for the erased case. The goal is to avoid iterating twice over the list (performance "gain").

Encode.string 4 ErasedType.Nothing // Encode.string could check if the value applied is marked with erased and in this case generate an empty JSON

I think adding Encode.erased this way would imply a lot of change internally to Thoth.Json and should probably be done in vNext branch as it is already a major rewrite of the library.

[Witold Szczerba]

Sometimes, the valid representation is just to exclude a field in JSON. It's not always a straight – one JSON field mapped by one domain type field – and vice versa. In my case, where I work with mixed environment with some legacy services, the domain modeling reveals big differences between domain objects and their JSON representation.

Let me show you one example:

type CustomerRequest =
    { Price: RequestedPrice
    // other fields }
and RequestedPrice =
    | LimitedOrder of decimal
    | MarketOrder

// remote service, accepting JSON:
{ "price_type": "FIXED", "price": "4.4365", /* other fields */ } // <--- the LimitedOrder case
{ "price_type": "AUTO", /* other fields */ } // <--- the MarketOrder case

This is why I would find it interesting to be able to create decoders which would produce a result of omitting the field entirely from a JSON response.

Also, imagine creating a JSON payload of MongoDB update. It's big difference between:
{ $set: { firstname: "…", lastname: null, somethingElse: null } } and
{ $set: { firstname: "…", somethingElse: null } }
First one will change firstname and erase both lastname and somethingElse, the latter will not alter lastname at all.

I could go one by one, example by example, where it does matter if the field is null vs it does not exist in an JSON object.

I thought it would be easy to filter out these erased or undefined when iterating over a list of tuples in Encode.object, but if that is not the case, then OK, there are workarounds.

[Maxime Mangel]

If you are interested in adding support for this feature only from Thoth.Json (Fable runtime) you could easily add it in your project using:

open Thoth.Json
open Fable.Core

module Encode =

    [<Emit("undefined")>]
    let jsUndefined : obj = jsNative

    let erased = jsUndefined

type ErasedType =
    | String of string
    | Nothing

    static member Encoder (v : ErasedType)=
        match v with
        | String str -> Encode.string str
        | Nothing -> Encode.erased

Encode.object [
    "name", Encode.string "Maxime"
    "x", ErasedType.Encoder Nothing
]
|> Encode.toString 4
|> printfn "%A"

REPL demo

I thought it would be easy to filter out these erased or undefined when iterating over a list of tuples in Encode.object, but if that is not the case, then OK, there are workarounds.

I don't think this is that simple because as I said Thoth.Json currently use 2 different JSON parser depending on the runtime. And currently, the Encode module directly manipulate JsonValue or JObject in the case of Newtonsoft.Net.

And thus object don't have a erased representation. We would have to instead make them manipulate a custom type from which we could identify the value marked as erased.

type EncodeValue = 
   | Keep of JsonValue // or Keep of JObject
   | Erased

Which means rewritting all the Encode module for that. That's why I said if I had it it will be in vNext which is already a complete rewrite of the library. (Almost done but just lacking the motivation right now to polish it always the last 10% which are the harder ^^)

[Maxime Mangel]

@njlr Do you think we should add the ability to omit fields ?

Do you think it is possible to do with the API proposed in #188 ? When using the JSON DUs, we probably could have added a new case to the DUs to cover this need.

[njlr]

This sort of thing also comes up in GraphQL, where a variable can have 3 states: something, nothing or absent.

My gut feeling is that this shouldn't be added because F# list comprehensions can contain match expressions. I'm open to good counter-examples though 🙂

IMO the example given above works well with a computation expression decoder.

#r "nuget: Thoth.Json.Net, 11.0"
#r "nuget: Thoth.Json.Net.CE, 0.2.1"

type CustomerRequest =
  {
    Price : RequestedPrice
    Foo : int
  }

and RequestedPrice =
  | LimitedOrder of decimal
  | MarketOrder

open Thoth.Json.Net
open Thoth.Json.Net.CE

module CustomerRequest =

  let encode : Encoder<CustomerRequest> =
    fun x ->
      [
        match x.Price with
        | LimitedOrder price ->
          "price_type", Encode.string "FIXED"
          "price", Encode.decimal price
        | MarketOrder ->
          "price_type", Encode.string "AUTO"

        "foo", Encode.int x.Foo
      ]
      |> Encode.object

  let decode : Decoder<CustomerRequest> =
    decoder {
      let! priceType =
        Decode.field "price_type" Decode.string

      let! price =
        match priceType with
        | "FIXED" ->
          Decode.field "price" Decode.decimal
          |> Decode.map LimitedOrder
        | "AUTO" ->
          Decode.succeed MarketOrder
        | _ ->
          Decode.fail $"Unexpected price_type `%s{priceType}`"

      let! foo = Decode.field "foo" Decode.int

      return
        {
          Price = price
          Foo = foo
        }
    }

// Quick tests...
let requests =
  [
    {
      Price = LimitedOrder 4.4365m
      Foo = 123
    }
    {
      Price = MarketOrder
      Foo = 456
    }
  ]

for request in requests do
  let json =
    request
    |> CustomerRequest.encode
    |> Encode.toString 0

  printfn $"%s{json}"

  let decoded =
    json
    |> Decode.unsafeFromString CustomerRequest.decode

  if request <> decoded then
    failwith $"Unexpected decoding: %A{decoded}"

Output:

{"price_type":"FIXED","price":"4.4365","foo":123}
{"price_type":"AUTO","foo":456}

[Maxime Mangel]

If needed we can revisit it later.