Database issues ..

[Steve Kemp]

We store our data in an SQLite file, and I've seen some failures this morning. Specifically the POST to /events is returning an error:

   May 26 23:13:26 www.steve.org.uk run[303]: 2018/05/26 23:13:26 
       http: Accept error: accept tcp 127.0.0.1:1044: accept4: too many open files; retrying in 1s

So looking at the process tree (metrics!) we see that we have a bazillion connections to our SQLite file open. We also have follow-on errors:

    Error reaping - database is locked

So I've made sure I'm calling Close() appropriately, and I've tweaked the DB open, but I suspect the actual issue is concurrency. There are two things that use the SQLite database:

• HTTP-based submission
• The processor-thread.

And reading the documentation here it shows that you must open different connections if you're in different goroutines. My processor-thread (invoked via the cron library ) is absolutely a different goroutine, behind the scenes.

So :

• We need to rework things to open a new connection.
• Or we need to use a different store.
  • SQL-based is ideal, but there's nothing too specific here.

[Steve Kemp]

Closed via the migration to MySQL - which allows us to run under the managed MySQL instance in the cloud.