tursodatabase / libsql

libSQL is a fork of SQLite that is both Open Source, and Open Contributions.

Home Page:https://turso.tech/libsql

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Insert queries take an enormous amount of time with replicas

ilshm opened this issue · comments

commented

tursoClient.js

import { createClient } from "@libsql/client"

const tursoClient = createClient({
  url: process.env.TURSO_DATABASE_URL,
  authToken: process.env.TURSO_AUTH_TOKEN,
})

export default tursoClient

app.js

import tursoClient from './lib/tursoClient.js'

async function testDB() {
    const iterations = 1000 // Number of iterations for stress testing

    // Variables to store performance metrics
    let totalQueryTime = 0
    let totalWriteTime = 0

    // Read operation
    for (let i = 0; i < iterations; i++) {
        const startQueryTime = Date.now()
        const employees = await tursoClient.execute("SELECT * FROM Employees")
        const endQueryTime = Date.now()
        totalQueryTime += endQueryTime - startQueryTime
    }

    // Write operation (inserting a new employee)
    for (let i = 0; i < iterations; i++) {
        const startWriteTime = Date.now()
        // Generate random data for the new employee
        const newEmployeeFirstName = await getRandomString()
        const newEmployeeLastName = await getRandomString()
        const newEmployeeDepartmentID = 1
        // Execute the INSERT query
        await tursoClient.execute({
            sql: "INSERT INTO Employees (FirstName, LastName, DepartmentID) VALUES (?, ?, ?)",
            args: [newEmployeeFirstName, newEmployeeLastName, newEmployeeDepartmentID]
        })
        const endWriteTime = Date.now()
        totalWriteTime += endWriteTime - startWriteTime
    }

    // Print performance reports
    const averageQueryTime = totalQueryTime / iterations
    console.log(`Average query time for SELECT operation: ${averageQueryTime} ms`)

    const averageWriteTime = totalWriteTime / iterations
    console.log(`Average time taken for INSERT operation: ${averageWriteTime} ms`)
}

async function getRandomString() {
  const characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
  let randomString = ''
  for (let i = 0; i < 5; i++) {
    randomString += characters.charAt(Math.floor(Math.random() * characters.length))
  }
  return randomString
}

await testDB()

Results:

Primary only:

root@ubuntu-s-4vcpu-16gb-amd-ams3-01:~/pastebin# node --env-file=.env app.js 
Average query time for SELECT operation: 20.641 ms
Average time taken for INSERT operation: 19.172 ms

Primary + 2 replicas:

root@ubuntu-s-4vcpu-16gb-amd-ams3-01:~/pastebin# node --env-file=.env app.js 
Average query time for SELECT operation: 13.753 ms
Average time taken for INSERT operation: 132.646 ms

Is this acceptable? Is there a way to disable replication acknowledgment for insert operations, similar to MongoDB?

Replication is async so it shouldn't impact the performance. This is something we will have to debug to understand why replication has such a big impact on performance.