Extreme compile times for read(data, Vector{CustomType})
fredrikekre opened this issue · comments
Trying to use JSON3 instead of JSON for https://github.com/fredrikekre/Canvas.jl and found that there are very long compile times in some cases. I tried to reduce it to just this file: https://gist.github.com/fredrikekre/f3f52938656c610b839c24033d54ba89 with the following results:
$ julia -e 'include("mwe.jl"); @time JSON3.read(array_data, Vector{Course}); @time JSON3.read(array_data, Vector{Course})'
35.656557 seconds (66.28 M allocations: 4.340 GiB, 4.40% gc time) <---- ??
0.000111 seconds (194 allocations: 10.219 KiB)
Reading to JSON3.Array
is fast, reading the same data (modulo []
) to JSON3.Object
is fast, and reading to CustomType
is at least reasonably fast:
$ julia -e 'include("mwe.jl"); @time JSON3.read(array_data); @time JSON3.read(array_data)'
0.936212 seconds (3.42 M allocations: 172.945 MiB, 8.15% gc time)
0.000039 seconds (13 allocations: 7.969 KiB)
$ julia -e 'include("mwe.jl"); @time JSON3.read(object_data); @time JSON3.read(object_data)'
0.935497 seconds (3.42 M allocations: 172.813 MiB, 8.37% gc time)
0.000013 seconds (8 allocations: 7.719 KiB)
$ julia -e 'include("mwe.jl"); @time JSON3.read(object_data, Course); @time JSON3.read(object_data, Course)'
5.611611 seconds (15.75 M allocations: 947.468 MiB, 9.85% gc time)
0.000100 seconds (184 allocations: 9.875 KiB)
The array_data is just of length 1 with the same object inside. Is there a better way to read data where you want Vector{CustomType}
back? Maybe just use JSON3.read()
and construct the vector myself from the resulting JSON3.Array
?
Anyway, I thought reading something to Vector{CustomType}
is common enought that it might be interesting to optimize or add special methods for.
Thanks for the report; let me try to dig in and see why the compile time is blowing up. Can I ask what julia version you're on for good measure?
FWIW, I see:
➜ dev julia -e 'include("/Users/jacobquinn/Downloads/json3perf.jl")'
1.870858 seconds (5.74 M allocations: 306.902 MiB, 8.61% gc time)
0.000122 seconds (188 allocations: 9.906 KiB)
on:
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.4.0-DEV.266 (2019-10-07)
_/ |\__'_|_|_|\__'_| | Commit d5d5718b85 (43 days old master)
|__/ |
On Julia 1.2, I see the performance hit you originally reported.
And for good measure, on v1.3.0-rc5, I see:
➜ dev julia -e 'include("/Users/jacobquinn/Downloads/json3perf.jl")'
1.858876 seconds (5.73 M allocations: 306.516 MiB, 9.30% gc time)
0.000103 seconds (188 allocations: 9.906 KiB)
So potentially, it's just a wait-for-1.3-to-be-officially-released kind of issue?
Okay yea I used 1.2, and didn't try on newer ones, sorry.
I see the same issue on Julia 1.4, very slow when reading into a vector or array of custom types.