OOM crash with too many files

Question

OOM crash with too many files

ferndot opened this issue 10 years ago · comments

Fern (they/them) commented 10 years ago

If there are a large number of files in a storage, Firetext crashes with Out Of Memory.

We need to:

Determine the point at which this problem starts.
Discuss mitigation strategies for this issue. (I believe that it might be fixed by #174)

Fern (they/them) · Answer 1 · Wed Nov 19 2014 22:16:54 GMT+0800 (China Standard Time)

Originally by @Ryuno-Ki. This is the directory structure of the storage that caused the crash:

$ du -h -d 2
3.8M    ./LOST.DIR
32K ./.android_secure
21M ./DCIM/100MZLLA
302M    ./DCIM/101MZLLA
3.5G    ./DCIM/MUSIC
1.6G    ./DCIM/VIDEO
21G ./DCIM/FILES
512K    ./DCIM/FFOS-Sticky-App-Imgs
128K    ./DCIM/FFOS-Drawl-App-Imgs
1.9G    ./DCIM/IMAGES
29G ./DCIM
66M ./.gallery/previews
67M ./.gallery
320K    ./.Trash-14908/info
25M ./.Trash-14908/files
26M ./.Trash-14908
128K    ./.Trash-1000/info
32K ./.Trash-1000/files
32K ./.Trash-1000/expunged
224K    ./.Trash-1000
32K ./downloads/bluetooth
64K ./downloads
5.5M    ./screenshots
64K ./.Trash-2111/info
3.8M    ./.Trash-2111/files
3.9M    ./.Trash-2111
64K ./contacts2xml/20130601020207
224K    ./contacts2xml
32K ./.downloaded
29G .

André Jaenisch · Answer 2 · Thu Nov 20 2014 06:59:08 GMT+0800 (China Standard Time)

Thanks, Hoshua,

some notes here:

32K is the size of every empty directory on the card's file system.
The output above contains only the upper most two levels (-d 2) Further details are too intrusive to me.
I filled a bug against Mozilla with ID #1099090 but no reaction by now.
I tried to reproduce the error in order to get a adb logcat to share, but … it worked.

You must have done some magic since I tried the app last.
Mostly .html and .txt files here.
Maybe it's clever to stress test the app in tests anyway …

André Jaenisch · Answer 3 · Thu Nov 20 2014 07:03:18 GMT+0800 (China Standard Time)

I'd like to emphasize that my bug against Mozilla is still valid (PDF Viewer and file browser continue crashing). Just your app is fine :)

Fern (they/them) · Answer 4 · Thu Nov 20 2014 10:03:57 GMT+0800 (China Standard Time)

@Ryuno-Ki: thanks for filing the bug. I will try to promote it so that it gets attention.

It is great to hear that Firetext didn't crash. We have done a bit of performance work lately =)

André Jaenisch · Answer 5 · Fri Nov 21 2014 18:32:55 GMT+0800 (China Standard Time)

@joshua-s Would you and your team like to share some knowledge with Mozilla Hacks?

As stated in the bugzilla bug, several apps suffer from getDeviceStorage(), so it would be helpful, to explain your approach. I can look after reaching out to Robert Nyman and point him to this issue.

Fern (they/them) · Answer 6 · Mon Nov 24 2014 22:57:23 GMT+0800 (China Standard Time)

@Ryuno-Ki I don't actually know what is preventing us from crashing. Our IO code is certainly very messy. Our performance improvements have been moving from SVG icons to a font, encapsulating the editor in an <iframe>, and optimizing some CSS rules.

If you would like to look, our IO enumerate function is here: https://github.com/codexa/firetext/blob/develop/scripts/io.js#L114

André Jaenisch · Answer 7 · Wed Nov 26 2014 22:15:45 GMT+0800 (China Standard Time)

FYI bug #1104337 was linked today.