Parsing before construct

[lsaenzt]

This is a question/feature request more than a issue. If this is not the right place to post let me know (maybe discourse...)

I am helping in a package that uses intensively JSON3 for parsing market data. Everything works perfect but the json source data has float numbers and dates as strings so we have to declare struct with no types (which I've read is bad for performance) and then change the field to the appopriate type (also bad).

Is there a way to avoid this and parse the data into the appropriate type before the field is updated (because it is #undef at that moment)? If not, could it be implemented?

Maybe the solution is in StructType.construct and StructType.keywords but I do not know how to make it work.

Thank you.

Luis

Here is an example of the code:

• Struct used for price data (with no types)

   mutable struct priceBucket
       price # float as string
       liquidity 

      priceBucket() = new()
   end

    mutable struct price
       type 
       instrument
       time # Date as string
       bids::Vector{priceBucket}
       asks::Vector{priceBucket} 
       closeoutBid # float as string
       closeoutAsk # float as string
       tradeable 

       price() = new()
   end

   JSON3.StructType(::Type{priceBucket}) = JSON3.Mutable()
   JSON3.StructType(::Type{price}) = JSON3.Mutable()

• Function we have to use to transform to the right type after the struct is populated

function coercePrice(price::price)`
    # Coerce Asks
    for ask in price.asks
        ask.price = parse(Float32, ask.price)
    end
    # Coerce Bids
    for bid in price.bids
        bid.price = parse(Float32, bid.price)
    end
    price.time = DateTime(first(price.time, 23), Dates.DateFormat("yyyy-mm-ddTHH:MM:SS.sssssssssZ"))
    price.closeoutBid = parse(Float32, price.closeoutBid)
    price.closeoutAsk = parse(Float32, price.closeoutAsk)

    return price
    end

[Jacob Quinn]

Hey @lsaenzt, really sorry about the delay here; I've been caught up in some stuff and trying to circle back around to things.

Yeah, this kind of scenario is a bit tricky, but I thought of a clever way we could get what you want here:

using StructTypes, JSON3
struct FloatAsString
    x::String
end
StructTypes.StructType(::Type{FloatAsString}) = StructTypes.StringType()
Base.string(x::FloatAsString) = x.x
StructTypes.construct(::Type{FloatAsString}, x::String) = Parsers.parse(Float64, x)

mutable struct priceBucket
    price::Union{FloatAsString, Float64} # float as string
    priceBucket() = new()
    priceBucket(price) = new(price)
end

StructTypes.StructType(::Type{priceBucket}) = StructTypes.Mutable()

pb = priceBucket(FloatAsString("3.14"))
json = JSON3.write(pb)
pb2 = JSON3.read(json, priceBucket)

The trick here is our custom FloatAsString type that is declared as StructTypes.StringType(). That means when JSON3.reading our priceBucket, it sees the Union{FloatAsString, Float64} and because FloatAsString is a StringType, it will parse a String first, then call StructTypes.construct(FloatAsString, "3.14"), we then are tricky in our definition of StructTypes.construct and parse it to a Float64 to return.

I'll keep thinking if there's a better way we could support something like this, but I think this should be pretty performant.

[lsaenzt]

Many thanks @quinnj . I will try this out.