[Bug] [Android] Out of memory error when uploading a large file
chrisjdev opened this issue · comments
Describe the bug
I was attempting to upload a ~300MB file using uploadFile() on Android 11. Very quickly it got several out of memory errors and eventually crashed. I traced the issue down to HttpRequest.copy(), where it will attempt to read the entire file, write the entire file, and HttpURLConnection will attempt to buffer the entire file in memory before sending, to have an accurate content-length. I realize this isn't a trivial problem, one not-great workaround is to use setChunkedStreamingMode(), but since that requires special server support, it's not an appropriate fix. Using setFixedLengthStreamingMode() would likely be appropriate, but figuring out what that should be with a multi-part upload doesn't sound trivial to me.
System info
- affected HTTP plugin version: 2.0.1
- affected platform(s) and version(s): Android 11
- affected device(s): Samsung S20
- cordova version: 10.0.0
- cordova platform version(s): android 9.0.0
Are you using ionic-native-wrapper?
- ionic-native-wrapper version: 5.22.0
- did you check ionic-native issue tracker for your problem? no
Minimum viable code to reproduce
N/A for now
Screenshots
N/A for now
Did you manage to fix the issue?
I'm afraid not. It looks like our customers only use iOS for the feature we needed it for.
I'm afraid not. It looks like our customers only use iOS for the feature we needed it for.
@chrisjdev You're able to upload files of 300MB in iOS? When we're trying to upload files larger than 10MB we're getting Failed to load resource: The operation couldn’t be completed. (WebKitBlobResource error 1.). Were you facing similar issues in iOS?
@pentanaveen Yes, uploading worked fine on iOS. Were you using the "uploadFile()" function? I think that's the key to avoiding attempting to load very large files into the JS VM and running out of memory.
@pentanaveen Yes, uploading worked fine on iOS. Were you using the "uploadFile()" function? I think that's the key to avoiding attempting to load very large files into the JS VM and running out of memory.
Oh! We were using post with FormData for uploading files. That might be the reason the request was failing for large files. I'll give uploadFile() a try. Thanks for pointing out.
setChunkedStreamingMode() need to be set.
For best performance, you should call either setFixedLengthStreamingMode(int) when the body length is known in advance, or setChunkedStreamingMode(int) when it is not. Otherwise HttpURLConnection will be forced to buffer the complete request body in memory before it is transmitted, wasting (and possibly exhausting) heap and increasing latency.
https://developer.android.com/reference/java/net/HttpURLConnection#posting-content
I tried to add the line below in CordovaHttpBase.java
request.chunk(0);
I got another error, can anyone help to fix it?
W/Cordova-Plugin-HTTP: Generic request error
com.silkimen.http.HttpRequest$HttpRequestException: java.net.SocketException: Connection reset
at com.silkimen.http.HttpRequest$Operation.call(HttpRequest.java:643)
at com.silkimen.http.HttpRequest.copy(HttpRequest.java:2502)
at com.silkimen.http.HttpRequest.part(HttpRequest.java:2801)
at com.silkimen.http.HttpRequest.part(HttpRequest.java:2771)
at com.silkimen.cordovahttp.CordovaHttpUpload.sendBody(CordovaHttpUpload.java:52)
at com.silkimen.cordovahttp.CordovaHttpBase.run(CordovaHttpBase.java:86)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:919)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
at com.android.okhttp.okio.Okio$1.write(Okio.java:78)
at com.android.okhttp.okio.AsyncTimeout$1.write(AsyncTimeout.java:157)
at com.android.okhttp.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:177)
at com.android.okhttp.okio.RealBufferedSink.write(RealBufferedSink.java:47)
at com.android.okhttp.internal.http.Http1xStream$ChunkedSink.write(Http1xStream.java:327)
at com.android.okhttp.okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:177)
at com.android.okhttp.okio.RealBufferedSink$1.write(RealBufferedSink.java:199)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at com.silkimen.http.HttpRequest$6.run(HttpRequest.java:2495)
at com.silkimen.http.HttpRequest$6.run(HttpRequest.java:2488)
at com.silkimen.http.HttpRequest$Operation.call(HttpRequest.java:637)
at com.silkimen.http.HttpRequest.copy(HttpRequest.java:2502)
at com.silkimen.http.HttpRequest.part(HttpRequest.java:2801)
at com.silkimen.http.HttpRequest.part(HttpRequest.java:2771)
at com.silkimen.cordovahttp.CordovaHttpUpload.sendBody(CordovaHttpUpload.java:52)
at com.silkimen.cordovahttp.CordovaHttpBase.run(CordovaHttpBase.java:86)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:919)
I/chromium: [INFO:CONSOLE(35)] "There was an error with the request: Connection reset", source: http://localhost/js/index.js (35)
I've been revisiting this issue a little bit. I'm guessing @NXTminxu is seeing a connection reset because something between writing the POST and the server interpreting the data doesn't like the chunked mode. More troubleshooting with that host would be needed to figure it out. I still think the key to resolving this issue is having a call to setFixedLengthStreamingMode() just as FileTransfer Plugin does here: https://github.com/apache/cordova-plugin-file-transfer/blob/master/src/android/FileTransfer.java#L418
I've been investigating alternatives. I was thinking FileTransfer Plugin could work as it did not have the oom issue, and apparently does progress updates. Unfortunately, I need to upload 2 files as part of the same multi-part request. I looked at capacitor-community/http. I believe it only does 1 file at a time, and likely has the oom issue (but I do not see one logged yet). I think the HttpRequest.java file is where the issue lives, so I looked up the original and found it has no changes. I was then looking for libraries that wrap HttpUrlConnection in an easier to use interface to replace HttpRequest. I found OkHttp should work, and I remembered it was already a dependency of Advanced HTTP. Unfortunately, it looks like that dependency was removed from the latest release. I think Apache HttpClient should also work and appears to be well maintained, but maybe adding another library dependency isn't a good idea for the project. I haven't had any luck getting the tests running. I'm thinking of changing HttpRequest.java to buffer the non-file data, and then add a new public method to finalize/send the request.
I made the changes here: chrisjdev@c58d86d If you want, I can create a pull request for it. I still haven't been able to get the automated tests working. I also changed my application to remove cordova-plugin-file-transfer (it was formerly deprecated, now it appears it's poorly maintained). That meant I needed progress to work (#88). I created a branch for that. I also needed keep-alive to work (#115 and #450) so I re-applied my change to keep the manager instance with the plugin. Once I had that in place, I was able to reproduce the issue in #197. I found it was setting a response serializer that would result in passing a NSData* to be sent through Cordova, which throws. I took that out and it didn't appear to have any ill effects. If it's needed for something, perhaps it could reset the serializer to a good one after the download completes. I then found that downloadFile on iOS again attempts to buffer the entire file in memory (and then copies it! for optimization!?!), so I switched OkHttp methods to avoid that issue. A few caveats...
• iOS upload progress appears to report that it uploads much faster than it's actually uploading. I think it's due to Apple buffering data in NSUrlSession, and I'm not sure there's an easy fix for it.
• iOS downloadFile appears to open a new socket connection, even though it should be able to use an existing socket connection from previous HTTP calls. I wasn't able to find much information about this, and I don't see any obvious causes. I'm guessing it's a NSURLSessionDownloadTask vs NSURLSessionDataTask issue and there would not be an easy fix. Android is able to use a single socket connection for all HTTP methods.
• I noticed some Authorization/Cookie headers being sent to one server, that may have be sent due to communications with a different server.
The changes are in this branch: chrisjdev@bc7e8d0 I tried to make the changes with as little impact to the public interface as possible to maintain compatibility with the Ionic wrapper.