"too many open files" error
smarj opened this issue · comments
I've been using skate on macOS for a few months, and recently moved to a local charm server running on Linux. I had around 400 items in a database that I exported with skate list. I've been importing them a couple of dozen at a time, and decided to import the rest today. I was getting a too many open files error. I upgraded to latest. The local charm server is now at 0.12.4 and skate is at 0.2.1. The error reporting has changed, but the result is still "too many open files".
Here is the (somewhat redacted) new error:
$ skate set ${KEY}@database.db "${VALUE}"
Error: Opening file: "/Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst" error: open /Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst: too many open files
unable to open: /Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst
github.com/dgraph-io/ristretto/z.OpenMmapFile
/home/runner/go/pkg/mod/github.com/dgraph-io/ristretto@v0.1.0/z/file.go:80
github.com/dgraph-io/badger/v3.newLevelsController.func1
/home/runner/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/levels.go:145
runtime.goexit
/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/asm_amd64.s:1571
Usage:
skate set KEY[@DB] VALUE [flags]
Flags:
-h, --help help for set
Opening file: "/Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst" error: open /Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst: too many open files
unable to open: /Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst
github.com/dgraph-io/ristretto/z.OpenMmapFile
/home/runner/go/pkg/mod/github.com/dgraph-io/ristretto@v0.1.0/z/file.go:80
github.com/dgraph-io/badger/v3.newLevelsController.func1
/home/runner/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/levels.go:145
runtime.goexit
/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/asm_amd64.s:1571
I have 367 keys in the database now, but the open file limit was set to 256. I ran ulimit -n 2048. It's almost like skate keeps recently added keys open, because I hadn't seen this before with more keys added slowly (3-10 per day) than the open file limit.
It's happening at the 2-5 new keys per day rate.
Sorry @bashbunni, but this was a long time ago. I moved to Redis because Skate wasn't working with the amount of keys I was creating. I don't even have any Charm stuff installed now.