"too many open files" error
jgoodall opened this issue · comments
When I run a reflex command within my $GOPATH, I get a bunch of errors related to "too many open files". Here is an example:
[info] Error while watching new path src/code.google.com/p/go.net/websocket: open src/code.google.com/p/go.net/websocket: too many open files
I am on Mac OS X 10.9. I installed with go get (and then moved the binary into my ~/bin). I can run reflex in other directories that only have a few files. Here is the reflex command I am running:
reflex -g '*.go' -- go test github.com/jgoodall/test
It actually doesnt seem to matter what the command is, I get the same errors.
Hi John! Thanks for your error report.
First I'll describe what the issue is, but skip to the end if you just want a workaround.
Reflex uses fsnotify, a cross-platform library for watching files. fsnotify does not provide a way to recursively watch directories (see howeyc/fsnotify#56), so in reflex I implemented a recursive watching mechanism. This has to traverse the whole tree underneath the level where you ask reflex to watch and recursively watch every directory in that tree for changes. If you're watching your whole $GOPATH, that's probably a lot of directories -- my machines all have several thousand.
You can check like this: find $GOPATH -type d | wc -l.
Mac OS X, by default, limits you tightly on the number of files you are allowed to open at once -- on my Mountain Lion MBP it's 2560. You can check with ulimit -a and look where it says "-n: file descriptors". For me, I was able to set it 10000 using ulimit -n 10000, and this made me able to watch my entire $GOPATH with reflex. (From reading about it here, here, and here, you might have to use launchctl. Google "mac os ulimit" and similar for more information.)
Doing this could possibly slow your system down and is basically a hacky workaround. There is a better way. On BSD-flavored systems, including Mac OS X, fsnotify uses kqueue. kqueue does not provide a way to efficiently watching directory trees. OS X does provide such a facility: it's FSEvents. There is an open issue at howeyc/fsnotify#54 to implement FSEvents support in fsnotify. Better, there is an initiative to include something like this package as os/notify right in the Go standard library for the next release, and some of the Go developers have shown interest in implementing recursive watches, possibly with FSEvents on Mac OS X, as part of that. Expect this to land within the next few months.
Finally -- is there a reason you need to watch your entire $GOPATH? Typically I run reflex in my project directory.
To summarize:
Solutions
- Run reflex in your project directory so it has fewer files to watch
- Use
ulimitorlaunchctlto raise the number of open files you are allowed - Wait several months for an FSEvents implementation to present itself
(Side note: you can watch the same issue for fsnotify: howeyc/fsnotify#35)
Ok, that makes sense. #1 works for me and that is what I am doing, but wanted to run it from my 'root' dir.
It would be nice if I could run it from one directory (e.g. GOPATH) but only have it watch for files in another directory, using a --directory "path/to/dir" option.
This issue has been significantly improved with the introduction of the --inverse-regex (-R) flag, which allows for ignoring files and directories.
Unfortunately, fsnotify is not progressing as quickly as anticipated and it doesn't look like an FSEvents implementation is likely to land soon. So for now, we're stuck with the current situation.
I've added a section to the README that lays out the options for dealing with this problem. I think that will have to do for now.
-R ".git" was the fix for me 👍