atom / watcher

Running 1000 parallel watchers on Windows eats about 10GB of RAM which feels really excessive.

Looking at the source, the first thing that jumps out at me is the pair of buffers in Subscription:

watcher/src/worker/windows/subscription.cpp

Lines 34 to 35 in 53a96b2

    
           buffer{new BYTE[buffer_size]}, 
        
           written{new BYTE[buffer_size]}

Both are currently 1MB, which is likely too high -- that's a lot of FILE_NOTIFY_INFORMATION structs -- and doesn't account for 10GB.

/cc @Arcanemagus who noticed this yesterday.

I think it was "only" 3 GB? It still seemed a bit excessive. I've also noticed that starting a polling watcher takes ~5 seconds to report back that it is active.

Ah, I misread your graph. 3 GB for 1000 watchers is still high, but more in line with what I'd expect from reading the source. Each watcher allocates a persistent 2MB for change notification buffers. While an event callback is active, I may be accidentally copying a buffer to get another 1MB? And 3MB * 1000 is roughly 3GB.

A few things that I can do to improve this:

Drop that buffer size to something more reasonable. 256KB maybe? The risk here is that smaller buffers are more likely to drop events from rapid notifications. I'll need to do some tuning to see how hard it is to trigger that on a fast SSD.
Locate and prevent that unnecessary buffer copy.

Oh, the picture I sent was total ~10 GB used, but that was system wide 😉.

	buffer{new BYTE[buffer_size]},
	written{new BYTE[buffer_size]}

Excessive RAM usage during parallel test