Possible memory leak?
pudquick opened this issue · comments
Disclaimer: I haven't run instruments yet to figure out where this is, just writing this up before I forget about it.
In a usage of Restor, the download of a 25GB .dmg file resulted in 45+ GB of memory being consumed during the process. I wasn't sure if the application was intentionally buffering to memory for some reason, but looking through
and the related code, I'm not seeing any intention to do this (also - nice trick hashing + downloading at the same time!)Would it be a side effect of the dispatch queue? Would the received byte chunks queue up in memory while it waited for disk writes?
Of note, the machine I was testing this on had 3x+ times the available space needed for holding the .dmg prior to starting, but as a result of the memory usage ended up filling the disk with vm swapfiles and eventually caused Restor to not be able to finish saving the file to disk. Yes, my fault for doing something like this on a disk with "low" available space, I know, but just something interesting / possible room for improvement somewhere.
If I get a chance to analyze it for leakage, I'll see if I can find where it's happening and update here.
We're missing an autoreleasepool in one of our dispatched blocks. I'll fix it when I get to the office tomorrow.
I just had a quick look at this and the place I thought we missed a pool had one.
I also can't re-create; we don't have any 25GB images but I tried downloading a 4GB image while monitoring the usage and it didn't go up significantly: it cycled between 29-31MB while the download proceeded. If you see this again, an Instruments capture would be interesting.
Can you provide some specifics on what you'd like me to test/measure while doing it? I'm not too familiar with macOS instrumentation on-demand, but I was able to reproduce an almost 15 minute wait with an unreal amount of memory on our ~25 GB download.
- Launch Restor and take it right up to the point that it would start downloading
- Launch Instruments.app, select the "Leaks" template, then "Choose"
- At the top under "Choose Target" it should show Restor in running applications, select it.
- Click the Record button and if necessary provide your password so Instruments can access process memory.
- Switch back to Restor, begin downloading.
- Stop the recording.
- Cmd+S, save the .trace file somewhere then compress it as it'll be huge, share it with us.
Alternatively, if you can reliably reproduce this, can you provide the exact steps and configuration you're using?
Also some details on replication that may lend itself to not being a bug in the code
We were able to replicate this issue based on network connection
- Over wifi? No problem
- Over ethernet with a physical embedded NIC (like a Mac Mini)? No problem
- MacBook Pro TouchBar with a USB-C ethernet adapter? Memory issue
I'm honestly not sure there's anything that you can do with regards to this, but I'll be interested to see what you make out of the trace file.
That's very helpful! Which specific USB-C adapter are you using? I've just tried with a Belkin one on a 13" Touchbar and can't get it to happen but it might be an adapter specific thing.
Also:
- Are you using the build of Restor from the releases page or did you build from source? If the latter, is it a release or debug build?
- Do you have any network kexts that might be intercepting data coming from the network? Next-gen AV junk, stuff like that?
Are you able to replicate with version 1.3?
We did not intentionally try and fix the leak, but did muck with the codepath it could be in...
https://github.com/google/restor/releases/tag/1.3
Any leaks?
Cool, closing for now.