[Enhancement]Background concurrent copying GC freed for sklearn model constrcutor in Java
momo1986 opened this issue · comments
Hello, dear guys.
Thanks for sharing.
I used your transferred framework with RandomForest and ExtraTreeClassifier.
It reports the memory leakage issue log.
I/zygote: Background concurrent copying GC freed 21(80KB) AllocSpace objects, 4(10MB) LOS objects, 49% free, 15MB/31MB, paused 5.015ms total 27.306ms
I/zygote: Background concurrent copying GC freed 50(79KB) AllocSpace objects, 6(26MB) LOS objects, 44% free, 29MB/53MB, paused 9.510ms total 38.731ms
I/zygote: Background concurrent copying GC freed 65(80KB) AllocSpace objects, 2(24MB) LOS objects, 29% free, 57MB/81MB, paused 19.260ms total 62.676ms
I/zygote: Background concurrent copying GC freed 35(63KB) AllocSpace objects, 3(40MB) LOS objects, 26% free, 65MB/89MB, paused 103us total 117.040ms
I/zygote: Background concurrent copying GC freed 120(80KB) AllocSpace objects, 3(64MB) LOS objects, 17% free, 113MB/137MB, paused 37.535ms total 187.472ms
I/zygote: Background concurrent copying GC freed 32(64KB) AllocSpace objects, 2(48MB) LOS objects, 20% free, 93MB/117MB, paused 79us total 102.969ms
I/zygote: Background concurrent copying GC freed 23(79KB) AllocSpace objects, 1(27MB) LOS objects, 16% free, 120MB/144MB, paused 99us total 204.801ms
I/zygote: Background concurrent copying GC freed 186(96KB) AllocSpace objects, 3(64MB) LOS objects, 20% free, 93MB/117MB, paused 66us total 144.466ms
I/zygote: Background concurrent copying GC freed 48(80KB) AllocSpace objects, 1(27MB) LOS objects, 16% free, 120MB/144MB, paused 107us total 191.767ms
I/zygote: Background concurrent copying GC freed 192(95KB) AllocSpace objects, 2(55MB) LOS objects, 20% free, 93MB/117MB, paused 394us total 114.586ms
I/zygote: Background concurrent copying GC freed 1037103(27MB) AllocSpace objects, 0(0B) LOS objects, 31% free, 52MB/76MB, paused 114us total 107.236ms
I/zygote: Background concurrent copying GC freed 1068277(28MB) AllocSpace objects, 0(0B) LOS objects, 29% free, 56MB/80MB, paused 116us total 144.632ms
I/zygote: Background concurrent copying GC freed 1120061(29MB) AllocSpace objects, 0(0B) LOS objects, 28% free, 60MB/84MB, paused 112us total 134.216ms
I/zygote: Background concurrent copying GC freed 1134345(30MB) AllocSpace objects, 0(0B) LOS objects, 27% free, 63MB/87MB, paused 133us total 138.526ms
I/zygote: Background concurrent copying GC freed 1165223(30MB) AllocSpace objects, 0(0B) LOS objects, 26% free, 68MB/92MB, paused 100us total 152.837ms
I/zygote: Background concurrent copying GC freed 1210424(32MB) AllocSpace objects, 0(0B) LOS objects, 25% free, 71MB/95MB, paused 91us total 167.858ms
I/zygote: Background concurrent copying GC freed 1225058(32MB) AllocSpace objects, 0(0B) LOS objects, 24% free, 75MB/99MB, paused 116us total 174.650ms
I/zygote: Background concurrent copying GC freed 1254940(33MB) AllocSpace objects, 0(0B) LOS objects, 22% free, 80MB/104MB, paused 91us total 189.392ms
I notice the constructor code will build Gson class to parse json.
public ExtraTreesClassifier (String file) throws FileNotFoundException {
String jsonStr = new Scanner(new File(file)).useDelimiter("\\Z").next();
Gson gson = new Gson();
Type listType = new TypeToken<List<Tree>>(){}.getType();
this.forest = gson.fromJson(jsonStr, listType);
this.nEstimators = this.forest.size();
this.nClasses = this.forest.get(0).classes[0].length;
}
Since sklearn model is stored in json, is there any method parse json while save the memory space?
Thanks & Regards!
Momo
Thanks for the hint @momo1986 ,
I noticed your issue and feature request. I will check whether it's possible to implement a lean solution.
But to be honest JSON isn't an optimal file format to store model data. gRPC is a way more better solution which I will focus on.
Kind regards,
Darius
Hello Guys,
I am getting this issue while working with bloc builder and bloc provider in a flutter. Memory is continuously leaking and even a simple reload taking a lot of time.
I/m.example.wavy(31983): Background concurrent copying GC freed 341886(19MB) AllocSpace objects, 0(0B) LOS objects, 49% free, 14MB/28MB, paused 817us total 330.645ms
I/m.example.wavy(31983): Background concurrent copying GC freed 346432(19MB) AllocSpace objects, 0(0B) LOS objects, 49% free, 14MB/28MB, paused 760us total 325.592ms
I/m.example.wavy(31983): Background concurrent copying GC freed 356566(19MB) AllocSpace objects, 0(0B) LOS objects, 53% free, 10MB/22MB, paused 327us total 123.724ms
I/m.example.wavy(31983): Background concurrent copying GC freed 240351(13MB) AllocSpace objects, 0(0B) LOS objects, 54% free, 10MB/22MB, paused 356us total 119.351ms
I/m.example.wavy(31983): Background concurrent copying GC freed 247576(13MB) AllocSpace objects, 0(0B) LOS objects, 52% free, 10MB/22MB, paused 508us total 202.753ms
I/m.example.wavy(31983): Background concurrent copying GC freed 251583(14MB) AllocSpace objects, 0(0B) LOS objects, 49% free, 12MB/24MB, paused 944us total 243.797ms
I/m.example.wavy(31983): Background concurrent copying GC freed 290476(16MB) AllocSpace objects, 0(0B) LOS objects, 49% free, 12MB/24MB, paused 329us total 222.792ms
I/m.example.wavy(31983): Background concurrent copying GC freed 286956(16MB) AllocSpace objects, 0(0B) LOS objects, 49% free, 13MB/26MB, paused 525us total 292.538ms
I/m.example.wavy(31983): Background concurrent copying GC freed 314817(17MB) AllocSpace objects, 0(0B) LOS objects, 49% free, 12MB/24MB, paused 954us total 243.960ms
I/m.example.wavy(31983): Background concurrent copying GC freed 286913(16MB) AllocSpace objects, 0(0B) LOS objects, 54% free, 10MB/22MB, paused 411us total 146.860ms
I/m.example.wavy(31983): Background concurrent copying GC freed 252150(14MB) AllocSpace objects, 0(0B) LOS objects, 56% free, 9MB/21MB, paused 404us total 111.100ms
I/m.example.wavy(31983): Background concurrent copying GC freed 237354(13MB) AllocSpace objects, 0(0B) LOS objects, 49% free, 12MB/25MB, paused 621us total 278.467ms
I/m.example.wavy(31983): Background concurrent copying GC freed 309332(17MB) AllocSpace objects, 0(0B) LOS objects, 50% free, 13MB/26MB, paused 657us total 289.556ms
I/m.example.wavy(31983): Background concurrent copying GC freed 334571(18MB) AllocSpace objects, 0(0B) LOS objects, 51% free, 11MB/23MB, paused 648us total 216.471ms
I/m.example.wavy(31983): Background concurrent copying GC freed 271849(15MB) AllocSpace objects, 0(0B) LOS objects, 56% free, 9MB/21MB, paused 1.542ms total 152.690ms
I/m.example.wavy(31983): Background concurrent copying GC freed 243302(13MB) AllocSpace objects, 0(0B) LOS objects, 53% free, 10MB/22MB, paused 1.293ms total 189.813ms
I/m.example.wavy(31983): Background concurrent copying GC freed 266205(14MB) AllocSpace objects, 0(0B) LOS objects, 56% free, 9MB/21MB, paused 466us total 143.352ms
I/m.example.wavy(31983): Background concurrent copying GC freed 242160(13MB) AllocSpace objects, 0(0B) LOS objects, 50% free, 12MB/24MB, paused 796us total 270.205ms
I/m.example.wavy(31983): Background concurrent copying GC freed 298920(16MB) AllocSpace objects, 0(0B) LOS objects, 50% free, 11MB/23MB, paused 692us total 265.460ms
I/m.example.wavy(31983): Background concurrent copying GC freed 299469(16MB) AllocSpace objects, 0(0B) LOS objects, 51% free, 11MB/23MB, paused 804us total 260.557ms
I/m.example.wavy(31983): Background concurrent copying GC freed 285154(16MB) AllocSpace objects, 0(0B) LOS objects, 58% free, 8MB/20MB, paused 767us total 161.538ms
I/m.example.wavy(31983): Background concurrent copying GC freed 239407(13MB) AllocSpace objects, 0(0B) LOS objects, 53% free, 10MB/22MB, paused 798us total 249.485ms
I/m.example.wavy(31983): Background concurrent copying GC freed 274651(15MB) AllocSpace objects, 0(0B) LOS objects, 52% free, 11MB/23MB, paused 803us total 257.307ms
I/m.example.wavy(31983): Background concurrent copying GC freed 284572(16MB) AllocSpace objects, 0(0B) LOS objects, 51% free, 11MB/23MB, paused 845us total 256.500ms
I/m.example.wavy(31983): Background concurrent copying GC freed 293973(16MB) AllocSpace objects, 0(0B) LOS objects, 55% free, 9MB/21MB, paused 559us total 206.293ms
I/m.example.wavy(31983): Background concurrent copying GC freed 262577(14MB) AllocSpace objects, 0(0B) LOS objects, 56% free, 9MB/21MB, paused 536us total 162.539ms
I/m.example.wavy(31983): Background concurrent copying GC freed 257412(14MB) AllocSpace objects, 0(0B) LOS objects, 58% free, 8MB/20MB, paused 734us total 146.732ms