skerkour / kerkour.com

(Ab)using technology for fun & profit. Programming, Hacking & Entrepreneurship @ https://kerkour.com

Home Page:https://kerkour.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Do not use `sqlx` for "high performance" sqlite insert benchmark

weiznich opened this issue · comments

As you try to optimize the time to insert a lot of items into a sqlite database, in your blog post here I thought it may be interesting for you to know that sqlx is the wrong tool for that job. Because of the "all is async" approach there is a quite large overhead for executing any query there. According to my own benchmarks executing queries using sqlx is 7 - 70 times slower than doing the same query using diesel. Using rusqlite normally gives you even faster queries. Therefore it might be worth to just use another database framework to improve the numbers there. See the benchmarks here for details on the numbers.

Disclaimer: As maintainer of diesel I might be biased there.

Hello,
Thank you very much for the feedback and your work on this great project!

From my experience, using an async database driver is the only way to manage complexity for any large project accessing a database, and sqlx does a great job at providing good balance between async, flexibility, typing and performance.

So it's certainly possible to speed up the thing, but it will be at the expense of simplicity, which is the main topic of the post.