Reading via a large number of goroutines breaks

[Anant Narayanan]

Reading from more than ~1000 goroutines simultaneously fails with an error "unable to open database file". Test program is attached. You can tweak maxRoutines to see the normal/broken behavior, on my local tests everything works fine until about 1000 goroutines.

package main

import (
    "database/sql"
    "fmt"
    "log"
    "math/rand"
    "os"
    "sync"

    _ "github.com/mattn/go-sqlite3"
)

const (
    maxKeys     = 20
    maxRoutines = 5000
)

func main() {
    // Create DB.
    os.Remove("./test.db")
    destDb, err := sql.Open("sqlite3", "./test.db")
    if err != nil {
        log.Fatal(err)
    }
    defer destDb.Close()
    destDb.Ping()

    // Create table.
    _, err = destDb.Exec("create table test(id int, value text)")
    if err != nil {
        log.Fatal(err)
    }

    // Write 20 values.
    for i := 0; i < maxKeys; i++ {
        _, err = destDb.Exec(fmt.Sprintf("insert into test values(%d, 'random_value')", i))
        if err != nil {
            log.Fatal(err)
        }
    }

    // Now test.
    stressTest(destDb)
    fmt.Printf("Success!\n")
}

func stressTest(db *sql.DB) {
    // Read values from all of these simultaneously.
    var wg sync.WaitGroup
    for i := 0; i < maxRoutines; i++ {
        wg.Add(1)
        go read(i, db, &wg)
    }
    wg.Wait()
}

func read(num int, db *sql.DB, wg *sync.WaitGroup) {
    defer wg.Done()
    _, err := db.Query(fmt.Sprintf("select value from test where id=%d", rand.Intn(maxKeys)))
    if err != nil {
        fmt.Printf("Goroutine %d failed read : %v\n", num, err)
    }
}

[mattn]

I got Success! with maxRoutines=5000.

[Anant Narayanan]

This may be a platform specific bug, I'm running on Mac OS X 10.11.5 (El Capitan)

$ uname -a
Darwin 15.5.0 Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64

[Allendar]

Yes it is. OS X limits OS-wide to not have more than 1000 files open simultaneously by default. This is also a very common issue when trying to use ApacheBench to test your Go server locally (also a reason to not use it anymore go Go server testing).

I think some Googling will likely get you to find how to adjust this limit based on your current OS X version. Take caution to tweak such things tho.

[Philip]

How come this stops working, when one uses prepared statements?