Copy file: progress monitoring and background capability

[ukudala]

User needs to have the application in foreground until video file has been copied to tmp directory. For a large file (i.e. >1.5GB), the copy of the file can take some time.

If the application enters background state, the copy is immediately cut off, and this process will be started over from scratch the next time application enters foreground.

Request

• How can I continue copying the file in the background?
• How can I monitor progress of file being copied?

---

Views of Vimeo.app on iOS, able to copy files in the background:
https://vimeo.com/user37760602/review/137998452/74a36e0d17
https://vimeo.com/user37760602/review/137998360/722c5db189

[ukudala]

http://stackoverflow.com/questions/4583012/nsfilemanager-continue-writing-to-disk-in-background

As this is changing things deep within VIMNetworking, would you be able to include this functionality?

I'll buy you a 🍺 and say thanks!! many times over. We are going to currently throw an alert to the user to tell them to stay in app till file copy completed; followed by another alert advising that they can leave the app upon completion. Definitely a sub-optimal path

[Alfie Hanssen]

@ukudala This is a great question, touches on some complex issues, and runs up against limitations of Apple's NSURLSession / background session APIs.

Seems like there are 2 primary use cases:

1. User initiates upload from within the app or an extension, app is in foreground when file copy starts, developer wants to ensure that user does not background the app during file copy, so we need a way to either communicate to the user that they need to wait, or we need a way to continue the copy operation beyond when the app is backgrounded.
2. User enqueues several uploads from within the app or an extension. User then backgrounds the app or finishes with the extension (assume copy operation is not interrupted). The upload system processes upload tasks serially. Upload system completes first upload task after app has been backgrounded or extension has been killed. App receives callback in AppDelegate that NSURLSession task finished. Upload system uses this opportunity to initiate second upload task, performing file copy wholly when app is in the background. This file copy must start and finish within the time allotted for an app to respond to the fact that an NSURLSession background task has completed. This is a finite, OS determined time window.

To my knowledge the only mechanism we have to influence this time window is via the UIApplication method - beginBackgroundTaskWithExpirationHandler: (which will not work for extensions) or the newer - performActivityWithOptions:reason:usingBlock: which should work for both.

I haven't tested either of these methods yet, and am not sure if they can extend the critical time periods mentioned above. My suspicion is that they can help in the case of #1 above but won't have any affect on #2 above.

I also think that there are files that are just too large for the current setup to handle. A 3GB file will almost certainly take more time than is naturally allotted, and more time than either of the above 2 methods can buy us. To accommodate this we could get into manually chunking the file.

So, where does this leave us?

I think your idea of alerting the user and asking them to remain in the foreground is a solid interim solution.

We track upload failures from the Vimeo iOS app closely. And only a very small percentage are due to interrupted file copy. There are a few higher priority failures. Which means that if we address the copy interruption it likely wont be for some time (we just don't have the bandwidth right now, with only 2.5 developers).

I also think this is just a limitation of the NSURLSession background session APIs. They're not designed for video upload, imo. If they were, Apple would allow us to upload directly from the ALAsset or PHAsset file without having the copy or export it to a temp directory. The need to copy completely cripples the power of the background task APIs. This of course is not an issue when dealing with images.

I'd like to keep this issue open though, because it would be an excellent refinement. Just got some bigger fish to fry first, hopefully that's understandable.

[ukudala]

I see the offer of a beer gets you going ;) While it would be prime to have VIMNetworking continue copy to tmp process in background, I agree with an alert presenting progress as an acceptable ux.

That being said, I have noticed some interesting behavior regarding upload completion of large files uploaded in background. Smaller files, naturally an easier task, will upload in the background and complete all saving processes. To test, I am able to view uploaded file(s) on Vimeo video page, with them beginning conversion.

Larger files, however, will upload to what I can only imagine as 100%, while not stopping short of required saving/completion processes. Using the same test as above, I will not be able to see the video posted for processing on Vimeo; only upon subsequently app run will this process be finished.

I find it odd that the background process would take care of the upload while stopping short of the final tasks. Bug? Possible to include this as a subsequent task when calling completionHandler?

I do think that the video's make it to Vimeo for processing however, two screenshots with timestamps illustrate the point:

at 8:19 it was 48 seconds ago

followed by:

8:24 saying 29 minutes ago

This certainly gives me the feeling that the video was received by Vimeo, but not completed until running the app again, which makes me think there is some required process that could fire but is currently not.

[Alfie Hanssen]

@ukudala

Beer yes, also upload yes, all cool stuff!

Just to make sure I understand what you're seeing. You are uploading a large file (how large?). You visit your vimeo.com My Videos webpage at some point after initiating the upload and backgrounding the app, and you see the first screenshot or the second? If you can help me understand the order of operations here and specifically why the timestamps look incorrect to you I can loop in some folks that work on the website portion of the upload system.

The other issue, regarding uploading large files from the app and feeling like they dont really stick until you open the app again. I think this can be explained by normal background session behavior. Background session tasks are what Apple calls "discretionary" when the app itself is in the background. Which basically means that the OS decides to process them when it feels like it, based on available / fluctuating system resources. Every upload processed by VIMNetworking is a process that involves 3 requests. If you initiate an upload and background the app. When the first request completes, the app will be alerted by the session process and given a chance to respond to the fact that that task completed (we respond by initiating the second request task). And repeat this process to initiate the third when the second completes. But initiating a task doesn't necessarily mean that the OS will execute it immediately. It may wait some time before doing so. I believe it may also pause / resume a task if necessary (like a long running file upload task).

My suspicion is that the behavior you're seeing is related to the file upload task completing in the background (the second request). And the third request being initiated but not yet started/completed by the OS for some time. Before it's started you are foregrounding the app at which point the system executes the task because it's no longer treated as "discretionary" when the app is in the foreground. So it completes immediately and you can see your upload on site.

There may of course be a bug here, but this expected background session behavior may also explain what you're seeing.

The only solid way I've seen to debug background session activity is by using local notifications. So you can run while not attached to the debugger (since this makes the background session perform a bit differently than in real life), but still have insight into what is happening within the upload system.

Check out the VIMTaskQueueDebugger class to enable local notifications and get a bit more insight into what's happening.

[Alfie Hanssen]

v2.0 of the upload library (to be released later this year) will require that users (developers) pass an accessible local url to the upload system (which means developers will be responsible for any upfront export or copy operations in advance). This is preferred > it's not really realistic to conduct copy in between background task completion / start, works fine for small files but will timeout for larger videos.