Possible memory leak, with reproduce repo

[wxsm]

Firstly, thanks for your awesome work on this driver.

Please see the reproduce: https://github.com/wxsms/go-sqlite3-memleak

in short:

1. I created a gin server
2. on each request in, I created a memory db, with a table created and some data inserted.
3. finally the db is closed

what's expect:

the memory should be released, since the db is close.

what's happening:

the memory keep grows and never released.

other infomations

it does not have to be a memory db, this problem also occurs in a regular file db. even if the tables are dropped, db file removed, the memory is still keep growing.

[mattn]

This is memory usage of small program below for 7 minutes.

package main

import (
	"database/sql"
	"fmt"
	_ "github.com/mattn/go-sqlite3"
)

func main() {
	f := func() {
		db, err := sql.Open("sqlite3", "file::memory:")
		if err != nil {
			panic(err)
		}
		defer db.Close()

		if _, err := db.Exec(`CREATE TABLE dummy_table (id INT PRIMARY KEY,f1 VARCHAR(50),f2 VARCHAR(50),f3 VARCHAR(50),f4 VARCHAR(50),f5 VARCHAR(50),f6 VARCHAR(50),f7 VARCHAR(50),f8 VARCHAR(50),f9 VARCHAR(50),f10 VARCHAR(50));`); err != nil {
			panic(err)
		}

		insert := "INSERT INTO dummy_table (f1,f2,f3,f4,f5,f6,f7,f8,f9,f10) VALUES "
		for i := 0; i < 9999; i += 1 {
			insert += fmt.Sprintf(`("%s","%s","%s","%s","%s","%s","%s","%s","%s","%s")`, "1111111111", "1111111111", "1111111111", "1111111111", "1111111111", "1111111111", "1111111111", "1111111111", "1111111111", "1111111111")
			insert += ","
		}
		insert = insert[:len(insert)-1]

		if _, err := db.Exec(insert); err != nil {
			panic(err)
		}
	}

	for {
		f()
	}
}

[mattn]

for 10 minutes.

[wxsm]

Thanks for response. Forgot to mention, I've tested on a web server, for requests with same table data, memory do not increase after a while, but it's not going down anyway. for requests with different table datas, memory keep growing and go OOM killed finally.

[wxsm]

Looks like for this simple program, it's mem usage stops at some point after a while. But I think the key problem in this case is that, the big part of memory cost is never getting gc. It's more obvious in a webserver. In a real production server of mime which accepts various table structure and data, it's getting OOM killed very fast.

If I remove the db insert usage, just create the big insert string and let go, the mem usage drops soon and keep at a low level about 10mb.

[hallgrenx]

@wxsms could it be that defer db.Close() never runs? Try move it before the r.GET() as the main function is never returning.

func main() {
   r := gin.Default()
   kdb, err := sql.Open("sqlite3", "file::memory:")
   if err != nil {
	panic(err)
   }
   defer db.Close()

   r.GET("/run", func(c *gin.Context) {
                 if _, err := db.Exec(`CREATE TABLE dummy_table (id INT PRIMARY KEY,f1 VARCHAR(50),f2 VARCHAR(50),f3 VARCHAR(50),f4 VARCHAR(50),f5 VARCHAR(50),f6 VARCHAR(50),f7 VARCHAR(50),f8 VARCHAR(50),f9 VARCHAR(50),f10 VARCHAR(50));`); err != nil {
			panic(err)
		}

		insert := "INSERT INTO dummy_table (f1,f2,f3,f4,f5,f6,f7,f8,f9,f10) VALUES "
		for i := 0; i < 9999; i += 1 {
			insert += fmt.Sprintf(`("%s","%s","%s","%s","%s","%s","%s","%s","%s","%s")`, "1111111111", "1111111111", "1111111111", "1111111111", "1111111111", "1111111111", "1111111111", "1111111111", "1111111111", "1111111111")
			insert += ","
		}
		insert = insert[:len(insert)-1]

		if _, err := db.Exec(insert); err != nil {
			panic(err)
		}
		c.String(200, "OK")
   })
   r.Run(":8080")
}