High memory consumption when loading `routes-large.kite`
minight opened this issue · comments
running routes-large.kite consumes an inordinate amount of memory, causing the program to start thrashing on machines with limited ram
The route compilation should be optimised to avoid this if possible
Profiling taken by curl localhost:6060/debug/pprof/heap > heap.pprof2 upon program startup with
./kr scan -w ~/routes-large.kite -A=apiroutes-210328:20000 -x 20 -j 1 --fail-status-codes 400,401,404,403,501,502,426,411 https://target.com seems to indicate that unmarshalling the kite file is consuming a substantial amount of memory.
I suspect using proto.Unmarshal is less optimal than using the inbuilt proute.ProtoAPI.Unmarshal function generated by gogoproto.
Later profiling indicates that converting to proto routes and then to http routes is causing 3x the memory usage necessary.
It may warrant rethinking how we load the dataset into memory and whether a non-copy version is possible
I have the same problem here.
