benchmarking

[jow blew]

I want to try to bench this with another sqlite implementation.

I got this though when i ran it:

x-MacBook-Pro:sdk apple$ go test
../../../../../../github.com/cznic/virtual/stdio.go:17:2: no buildable Go source files in /Users/apple/workspace/go/src/github.com/cznic/ccir/libc/errno
../../../../../../github.com/cznic/virtual/pthread.go:12:2: no buildable Go source files in /Users/apple/workspace/go/src/github.com/cznic/ccir/libc/pthread
../../../../../../github.com/cznic/virtual/stdio.go:18:2: no buildable Go source files in /Users/apple/workspace/go/src/github.com/cznic/ccir/libc/stdio
../../../../../../github.com/cznic/sqlite/sqlite.go:66:2: no buildable Go source files in /Users/apple/workspace/go/src/github.com/cznic/sqlite/internal/bin
x-MacBook-Pro:sdk apple$ 

The code:

package main

import (
	"database/sql"
	"fmt"
	"log"
	"os"

	// _ "github.com/mattn/go-sqlite3"

	_ "github.com/cznic/sqlite"
)

// Rundb does ...
func Rundb() {

	// pump log into "t3"

	os.Remove("./foo.db")

	db, err := sql.Open("sqlite3", "./foo.db")
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	sqlStmt := `
	create table foo (id integer not null primary key, name text);
	delete from foo;
	`
	_, err = db.Exec(sqlStmt)
	if err != nil {
		log.Printf("%q: %s\n", err, sqlStmt)
		return
	}

	tx, err := db.Begin()
	if err != nil {
		log.Fatal(err)
	}
	stmt, err := tx.Prepare("insert into foo(id, name) values(?, ?)")
	if err != nil {
		log.Fatal(err)
	}
	defer stmt.Close()
	for i := 0; i < 100; i++ {
		_, err = stmt.Exec(i, fmt.Sprintf("こんにちわ世界%03d", i))
		if err != nil {
			log.Fatal(err)
		}
	}
	tx.Commit()

	rows, err := db.Query("select id, name from foo")
	if err != nil {
		log.Fatal(err)
	}
	defer rows.Close()
	for rows.Next() {
		var id int
		var name string
		err = rows.Scan(&id, &name)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println(id, name)
	}
	err = rows.Err()
	if err != nil {
		log.Fatal(err)
	}

	stmt, err = db.Prepare("select name from foo where id = ?")
	if err != nil {
		log.Fatal(err)
	}
	defer stmt.Close()
	var name string
	err = stmt.QueryRow("3").Scan(&name)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(name)

	_, err = db.Exec("delete from foo")
	if err != nil {
		log.Fatal(err)
	}

	_, err = db.Exec("insert into foo(id, name) values(1, 'foo'), (2, 'bar'), (3, 'baz')")
	if err != nil {
		log.Fatal(err)
	}

	rows, err = db.Query("select id, name from foo")
	if err != nil {
		log.Fatal(err)
	}
	defer rows.Close()
	for rows.Next() {
		var id int
		var name string
		err = rows.Scan(&id, &name)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println(id, name)
	}
	err = rows.Err()
	if err != nil {
		log.Fatal(err)
	}
}

[cznic]

From the documentation

Supported platforms and architectures

See http://github.com/cznic/ccir. To add a newly supported os/arch combination to this package try running 'go generate'.

Following the links we finally get

Supported platforms and architectures

In GOOS_GOARCH form

linux_amd64
linux_386

If you can access a machine with a not yet supported os/arch and you would like to contribute to porting this package, you may want to start by trying

$ cd $GOPATH/github.com/cznic/ccir/libc
$ go generate

Please fill an issue for the port and let's discuss it there.

---

We are looking for developers and maintainers of not yet supported ports. It would be definitely nice to support Darwin. Do you perhaps want to try to contribute the port? In this case it is possibly much easier than the effort required to create the Windows port (which is now 99.9% ready thanks to @steffengy).

In any case, you would not like the numbers any benchmark can give you now. This project ATM interprets the sqlite C code in a VM. I guess being slower than native code at least a hundred times is what one can reasonably expect. With some heavy optimization it can maybe be lowered to only one order of magnitude slower. What I'm trying to point out is that the current VM implementation helps mainly in preparing a reliable toolchain and a platform for the final goal: transforming the IR code into Go code. Only then can be the driver expected to perform comparably to via cgo called sqlite driver.

Anyway, for some applications even the VM implementation can be possibly useful already. And some optimizations are already being developed and tested. There is some low hanging fruit around.

[Steffen Butzer]

As a side node:

For only running SQLite it MIGHT be enough
to just reuse the linux build by copying internal/bin/bin_linux_amd64.go to internal/bin/bin_darwin_amd64.go.
This might work since the mac and linux kernel are atleast similar, but no guarantees.

[jow blew]

@cznic @steffengy

thanks for the help ! Am out of my depth but what the hell :)

trying out your tips.

1. I made a "$GOPATH/src/github.com/cznic/sqlite/internal/bin/bin_darwin_amd64.go" with a copy from the internal/bin/bin_linux_amd64.go

2. Then the go generate for $GOPATH/src/github.com/cznic/ccir:

x-MacBook-Pro:libc apple$ pwd
/Users/apple/workspace/go/src/github.com/cznic/ccir/libc
x-MacBook-Pro:libc apple$ go generate
../../virtual/stdio.go:17:2: no buildable Go source files in /Users/apple/workspace/go/src/github.com/cznic/ccir/libc/errno
../../virtual/pthread.go:12:2: no buildable Go source files in /Users/apple/workspace/go/src/github.com/cznic/ccir/libc/pthread
../../virtual/stdio.go:18:2: no buildable Go source files in /Users/apple/workspace/go/src/github.com/cznic/ccir/libc/stdio
libc.go:1: running "go": exit status 1
x-MacBook-Pro:libc apple$ 

Then i went further down this rabbit hole.

$GOPATH/src/github.com/cznic/ccir
seems to be used and in there there are no darwin files so i guess it fails at that level ?
But i saw that the files inside the libc folder are GOARCH specific, so how to generate darwin ones. ?? hmmmm.

like i said out of my depth on this one. :)

Non of you have an OSX machine ?

[jow blew]

I make this issue, since i now get that its dependent on this:

cznic/ccir#3

[Steffen Butzer]

I made a "$GOPATH/src/github.com/cznic/sqlite/internal/bin/bin_darwin_amd64.go" with a copy from the internal/bin/bin_linux_amd64.go

This won't fix go generate for ccir.
This will only allow you to build sqlite (by using go build/go test there,
since you skip the process the bin file is usually created)

[cznic]

@joeblew99 Plase see my reply at cznic/ccir#3.

[cznic]

I made a "$GOPATH/src/github.com/cznic/sqlite/internal/bin/bin_darwin_amd64.go" with a copy from the internal/bin/bin_linux_amd64.go

FTR: I doubt that can work. Even if all of the required syscalls may exist on both platforms, their numbering differ sometimes even between different architectures of the same OS.

[Steffen Butzer]

FTR: I doubt that can work. Even if all of the required syscalls may exist on both platforms, their numbering differ sometimes even between different architectures of the same OS.

I do too, just saying it's possible with a bit of luck

[cznic]

You're right. It's very easy to try the luck that way. But I would be scared to potentially call random syscalls on my machine ;-)

[jow blew]

@cznic shoudl this be closed now, since it open cznic/ccir#3.

[cznic]

It can be closed, but we can also keep it open until the OP can run his benchmarks on Darwin. I have no strong opinion here.

[jow blew]

adios then. Benchmarking is not the issue, but darwin is.