Android memory leak due to JSC
ben-manes opened this issue · comments
related to: facebook/react-native#23259; cc @akshetpandey
Issue Description
A full GC is performed only when Heap#overCriticalMemoryThreshold returns true or if the default is changed to disable the generational GC. Otherwise only the young generation is collected. Unfortunately, overCriticalMemoryThreshold is only implemented for iOS and always returns false for other platforms. This means that any objects promoted from young to old are not GC'd and the app will eventually crash due to an out of memory error.
Please note that running RN through the Chrome debugger does not use JSC. This will use Chrome's JS engine and promptly reclaim memory. We confirmed this bug and fix by reducing the maximum memory for JSC, observing the leak, and then healthy behavior when resolved.
Version, config, any additional info
The least invasive change was to force a full collection and we are satisfied with the resulting performance. In our application, we did not experience long pause times that would result in a negative experience. We changed overCriticalMemoryThreshold to return true on non-iOS platforms and believe that is the correct default behavior. A more advanced solution would be to add Android support to calculate if the threshold was crossed.
iff --git a/webkit/Source/JavaScriptCore/heap/Heap.cpp b/webkit/Source/JavaScriptCore/heap/Heap.cpp
index ffeac67d..9dcf4113 100644
--- a/webkit/Source/JavaScriptCore/heap/Heap.cpp
+++ b/webkit/Source/JavaScriptCore/heap/Heap.cpp
@@ -498,7 +498,7 @@ bool Heap::overCriticalMemoryThreshold(MemoryThresholdCallType memoryThresholdCa
return m_overCriticalMemoryThreshold;
#else
UNUSED_PARAM(memoryThresholdCallType);
- return false;
+ return true;
#endif
}We substitute the android-jsc dependency with our patched version,
// replace the webkit jsc package with react native with our customized only do full
// garbage collection to avoid the memory leak issue in js heap
configurations.all {
resolutionStrategy.dependencySubstitution {
substitute module('org.webkit:android-jsc') with module('com.withvector:android-jsc:r225067')
}
}
Thanks @ben-manes for your report. We discussed this issue internally on contributors discord and with help from some jsc maintainers figured one other approach to fix this. We are hoping to find some time to investigate it and patch soon
@ben-manes I am trying to deal with the issue. Do you mind to share how you profile the memory usage? Before solving the issue, I would like to write some JS test cases to show the leakage in current JSC. But unfortunately, it seems I cannot force JS allocation marked as old object.
Here are my test case:
class GCTestCase extends React.Component<{}> {
state = {
counter: 0,
};
componentDidMount() {
global.buckets = [];
this.timer = setInterval(() => {
if (this.state.counter >= 1000 * 60 * 1) {
global.buckets = [];
this.setState({counter: 0});
} else {
global.buckets.push(new ArrayBuffer(1024));
this.setState(state => ({counter: state.counter + 1}));
}
}, 1);
}
componentWillUnmount() {
if (this.timer != null) {
clearInterval(this.timer);
this.timer = null;
}
}
render() {
return (
<View
style={{
flex: 1,
justifyContent: 'center',
alignItems: 'center',
backgroundColor: 'blue',
}}>
<Text style={{fontSize: 32, color: '#ffffff'}}>
${this.state.counter}
</Text>
</View>
);
}
}And check the memory usage with Android Studio memory profiler or adb shell dumpsys meminfo.
Hopefully if there are some test cases and could prove fixes works.
Thanks for taking a look at this! We used adb to investigate with since Android Studio's profiler didn't work with Android 4.4, iirc. We were able to test on a memory constrained device, which made it easier to use our application and monitor the memory report.
We'll try to get some notes together and review our steps. We did this last year during a scramble, as it blocked a partnership, so I am a little foggy at the moment.
Thanks @ben-manes,
Do you remember more information about the app, will the app being killed by Android after long run?
JSC's memory model is complicated, AFAIK, JSC will do full GC if eden / old ratio is too small.
Please see https://trac.webkit.org/browser/webkit/releases/WebKitGTK/webkit-2.22.2/Source/JavaScriptCore/heap/Heap.cpp#L2263.
From this, the memory usage may not as good as JSC on iOS, but it should not cause long running memory leakage.
We have it failing in our app with logs, and an early failure with yours. We're wrapping up a test case for you. The main observation is that we used 50 * 1024 * 1024 whereas you are using 1024, which will take much longer to exhaust. The larger arrays will be more aggressively promoted and seems to do the trick for us. I hope to post a sample based on your in a few hours.
@Kudo We modified your code to increase the interval from 1ms to 5s and also increased the buffer from 1K to 50MB. So every 5s, allocates 50MB of buffer and after 250MB deallocates everything. We tested this on Android 8 (API 26).
- The default.jsc contains the logs for default jsc that comes with react native 0.58.3.
- The
custom.full.gc.jscuses are patched version (above)
android8.api26
├── custom.full.gc.jsc
│ ├── logcat.txt
│ └── memory.usage.log.txt
├── default.jsc
│ ├── logcat.txt
│ └── memory.usage.log.txt
└── diff.txt
Awesome @ben-manes !
Thank you for the useful information you provided and I did see the leakage from your log.
May I further ask your device model to known which ABI you used and how many RAM you have?
I will try to see if I could reproduce and troubleshoot the leakage with your 5s / 50MB allocation method.
I've tested on Samsung Galaxy Note 5 (Android 7, arm32, 4GB RAM) without the leakage issue.
adb result range from 300MB - 900MB in total and will recycle memory.
Note that JSC by default will enable concurrent GC only for 64-bit devices.
#if !CPU(X86_64) && !CPU(ARM64)
Options::useConcurrentGC() = false;
#endif
Not sure the concurrent GC will affect the leakage or not.
It was a moto-e armabi-v7a, not sure about the model & ram until I get to the office
The attached tests were on a moto e5 plus 2gb.
Thanks for your information.
I was wrong that my Galaxy Note 5 is arm64-v8a as well.
Right now I have no idea why my device cannot reproduce the leakage.
Will turn another way to implement the missing part of bmalloc::availableMemory() for Android.
This enables us to have overCriticalMemoryThreshold() like iOS did.
Hope that will help the leak issue.
I have access to a whole bunch of android devices and will run the tests on them sometime in the next 24 and let you all know what the results are.
@ben-manes : Can you tell me how to create the gc profiles?
@akshetpandey I think you want adb shell dumpsys meminfo <package>
while true; do date >> log.txt; adb shell dumpsys meminfo <package> >> log.txt ; sleep 6; done
Perfect thanks, and I am assuming running a build with 0.59 rc and 0.57 should be a good comparison?
Yes. We recently upgraded from 0.53 to 0.58.3, and the above was verified on the later version. We haven't tested 0.59-rc, but there is no reason to expect it to work since we saw the issue in a relatively recent JSC build.
afaict, 0.85 doesn't use the new JSC from https://github.com/react-native-community/jsc-android-buildscripts/ but 0.59 does. I am unsure as to why you will be seeing the issues on 0.58 unless you are using a different jsc than the once bundled by default.
If you're still in SF and can't reproduce it, you are welcome to drop by our office and we'll show you the bug.
I haven't been able to repo it on 0.59 using a pixel 3, but I will try other devices. Here are the repo projects I am using. As far as I understand it, there should be a memory leak on 0.59 but not on 0.58.
leak_app_058.zip
leak_app_059.zip
It uses the GCTestCase class with 50 MB allocations and 5 second interval.
I am checking memory usage using:
watch -t 'adb shell dumpsys meminfo --local -s com.leak_app_059 | grep TOTAL | gsed -E "s/[^0-9]*([0-9]+).*/\1/"'One thing that seems to be wrong is, allocation 250 MB isn't reflected in meminfo, but changing the code to fill in the array:
const buffer = new ArrayBuffer(allocationSize);
const array = new Uint32Array(buffer);
array.fill(0xAAAAAAAA);
global.buckets.push(buffer);The memory usage finally shows up in meminfo, but I am still not seeing a leak. Total usage spikes pretty high but comes down to a reasonable value.
Results. Still have to investigate whats going on with Galaxy Nexus and if it repos on 0.58.
It is very possible that the leak exists and that the test case as it is right now doesn't work. I will have to look into to more to figure out how to force allocation to be moved to old gen.
| Device | ABI | API | Crashes | Leaks | Baseline Memory Usage |
|---|---|---|---|---|---|
| Pixel 3 | arm64-v8a | 28 | No | No | 235 MB |
| Pixel 3 XL | arm64-v8a | 28 | No | No | 255 MB |
| Pixel XL | arm64-v8a | 28 | No | No | 235 MB |
| Nexus 6P | arm64-v8a | 26 | No | No | 340 MB |
| Nexus 5X | arm64-v8a | 23 | No | No | 255 MB |
| Nexus 5 | armeabi-v7a | 19 | No | No | 120 MB |
| Nexus 7 | armeabi-v7a | 22 | No | No | 220 MB |
| Galaxy Nexus | armeabi-v7a | 18 | Yes | ?? | ?????? |
| Galaxy S9 | arm64-v8a | 26 | No | No | 340 MB |
| Galaxy S6 | arm64-v8a | 23 | No | No | 320 MB |
| Galaxy S5 | armeabi-v7a | 23 | No | No | 200 MB |
| Galaxy On5 | armeabi-v7a | 23 | No | No | 35 MB |
| Galaxy J1 | armeabi-v7a | 23 | No | No | 70 MB |
| Zenfone 5 | x86 | 19 | No | No | 140 MB |
| Zenfone 2 | x86 | 21 | No | No | 200 MB |
| HTC Desire 530 | armeabi-v7a | 23 | No | No | 165 MB |
| Oppo F1 | arm64-v8a | 22 | No | No | 110 MB |
In our tests the leak appears under Unknown memory in our memory logs. I don't think there is anything special about our usage that should differ, so not sure why you can't reproduce it.
I can clearly see the increase in TOTAL memory usage it 50MB steps and then a big dip whenever the gc runs. The GC runs don't align with when the array gets cleared.
Its likely the test case doesn't work and the memory is not being put into old gen.
Hi @ben-manes,
Just to double confirm that your android8.api26.log.tar.gz results are from pure test case JS code.
I.e. App.js includes only <GCTestCase> without any other code.
For default.jsc, did you test on RN 0.58 with new JSC from npm?
Otherwise, as @akshetpandey said, RN 0.58 in stock JSC is still old one.
@akshetpandey BTW, your test results from variant devices is really helpful to me, thank you so much.
We ran the test in a view of our app. We didn’t create a new app to run it in isolation.
Hopefully if you could test in a pure app and make sure we are on the same page.
I am going through the jsc code and trying to figure out how allocation can be forced into old gen. But it doesn't seem to be straightforward. JSC's gc changed significantly between the two versions. More here: https://webkit.org/blog/7122/introducing-riptide-webkits-retreating-wavefront-concurrent-garbage-collector/
The attached result are based on the leak_app_058/059.zip test app. We did a ./gradlew assembleRelease and install the apk on the Moto E5 Plus running Android 8.0.0 with about 3GB of RAM. On the leak_app_058, it was reproducible pretty easily when the memory usage reaches about 1.5GB and on the leak_app_059, it spikes up to 900MB but comes down to 200MB and did not crash. Please remember that you cannot attach the Chrome debugger, which is why we install the APK instead.
We forked at r225067 at commit 5fcba1a on Oct. 25th. It appears that 0.59 is on r236355.1.1 which does not crash in the sample app. However, it does appear to reclaim slower than our fix due to amortizing the clean up every ~200mb of garbage, whereas ours forces a full GC every cycle so it is immediate.
We still think the default behavior of overCriticalMemoryThreshold should default to true for unsupported platforms, as a safer fallback in case of a regression.
Thanks for your help to test the sample directly.
The result seems a little weird to me.
leak_app_058 uses the old JSC from 4 years ago.
RN 0.58 also support Android 64. For arm64/x64 version JSC, it comes from https://github.com/gengjiawen/android-jsc/tree/feature/abi_support.
If you have the memory leakage in 0.58, you may also have the same issue from RN 0.4x - 0.57, as RN uses the old JSC for a very long time.
Is my understanding correct that you have the leak issue for a very long time, from RN 0.4x - 0.57?
@Kudo Yes. When we found the leak, we were on 0.53. We had some crash reports (from Crashlytics) regarding memory, but rare and nothing we could track down. We went through a QA process for a partnership to be on the PCT tablet, which is an MSI MS5 Android 4.4.1 device (used in a large chunk of the trucking market). In this case our app would be open for a long time and this device only has 1gb of memory. Since it was blocking a lot of sales, we scrambled to determine the cause and fix it. We didn't have the bandwidth to report it and assist until now.
If that's the case, there is a chance that 0.59 fixes the problem. Though the spike in memory usage will be higher than your fix of always saying its over critical memory threshold.
@akshetpandey yes, that was what we saw today when running your sample app on 0.59 (see above)
Thank you @ben-manes, it is really terrible that old JSC had this memory leakage for a very long time.
And thankfully new JSC r236355.1.1 does not have the problem.
I will continue implement the overCriticalMemoryThreshold like iOS and memory usage could be better than now, but I still need some time from my rare free time slot.
At least the issue should not be a blocker for RN 0.59 release.
Yes, thank you @Kudo and @akshetpandey for looking into this. We didn't realize it was fixed with 0.59, and thought we had tried a recent version of jsc but still had to patch. I can close these issues whenever you want.
I had similar issues after upgrading to 0.59.x (but not sure if the problems existed before upgrade). Anyways RN 0.59.10 did not solve the issue even though JSC is updated in that version.
But then I ran into this: https://www.npmjs.com/package/react-native-v8
A drop-in replacement for JSC using V8. By quick testing it really seems to just work just by installing and adding those few lines to build scripts.
And best of all it solved memory issues!
Blog post about all the different JS engines: https://dev.to/anotherjsguy/react-native-memory-profiling-jsc-vs-v8-vs-hermes-1c76
@sakarit we change js engine to v8, but seems not work. the "Unknown" memory always grow
it looks like this is same as the problem that i am dealing with. I have also seen that Facebook app crushes due to memory leak (if u leave the app open).
I cant figure out how to make this change. where is this file? how can i change it? how can i compile it with the new value? i will be glad for some instructions
We face to the same problem on 0.59.10. The "Unknown/Private Other" part memory keeps growing.