Congratulations and benchmarks project
frederikhors opened this issue · comments
My happiest congratulations on this project! Very good!
Can I ask you for an opinion on this benchmark project?
What do you think about adding dbq too?
It's very easy, just copy one of these files and edit it with dbq code.
I found dbq today and I still need to understand it.
Again, thanks for your amazing project!
Thanks for the compliment.
I won't have time but here is a sample of how to benchmark dbq
https://medium.com/@rocketlaunchr.cloud/how-to-benchmark-dbq-vs-sqlx-vs-gorm-e814caacecb5
You should be able to add it easily. Dbq is really just a an extension to database/SQL - not an orm (yet, but v3 will be)
Let me tag the tip first because I believe there is some changes in master that have not been tagged
Can you please help me with this?
I'm trying to add dbq to this benchmark project: https://github.com/frederikhors/orm-benchmark/tree/add-dbq.
But I'm getting this error:
dbq
pq: syntax error at or near ","
Why?
Here the relevant code: https://github.com/frederikhors/orm-benchmark/blob/add-dbq/benchs/dbq.go
Thanks.
https://godoc.org/github.com/rocketlaunchr/dbq/v2#INSERTStmt
add dbq.PostgreSQL as final argument to INSERTStmt. The default is mysql.
Model will also need to implement https://godoc.org/github.com/rocketlaunchr/dbq/v2#ScanFaster for maximum performance (when querying/unmarshaling)
Thanks. Updated.
Now the error is pq got 8 parameters but the statement requires 7.
I think I don't have to add "id" in columns, right?
Relevant code: https://github.com/frederikhors/orm-benchmark/blob/add-dbq/benchs/dbq.go.
type Model struct {
Id int `orm:"auto" gorm:"primary_key" db:"id" dbq:"id"`
Name string `dbq:"name"`
Title string `dbq:"title"`
Fax string `dbq:"fax"`
Web string `dbq:"web"`
Age int `dbq:"age"`
Right bool `dbq:"\"right\""`
Counter int64 `dbq:"counter"`
}
stmt := dbq.INSERTStmt("models", []string{"id", "name", "title", "fax", "web", "age", "\"right\"", "counter"}, 1, dbq.PostgreSQL)
Done. Now the error is:
pq: duplicate key value violates unique constraint "models_pkey".
That means that you have a unique key in the table + you are inserting the same value twice.
Are you deleting all rows for each package you are benchmarking?
This is not a dbq issue.
I'm using this code for each orm:
for i := 0; i < b.N; i++ {
m.Id = 0
stmt := dbq.INSERTStmt("models", []string{"id", "name", "title", "fax", "web", "age", "\"right\"", "counter"}, 1, dbq.PostgreSQL)
_, err := dbq.E(context.Background(), dbqdb, stmt, nil, dbq.Struct(m))
if err != nil {
fmt.Println(err)
b.FailNow()
}
}Each orm works creating a new row with progressive ID.
Dbq gives that error.
Am I wrong?
ahhh i see.
type Model struct {
Id int `orm:"auto" gorm:"primary_key" db:"id" dbq:"-"` // replace with dash
Name string `dbq:"name"`
Title string `dbq:"title"`
Fax string `dbq:"fax"`
Web string `dbq:"web"`
Age int `dbq:"age"`
Right bool `dbq:"\"right\""`
Counter int64 `dbq:"counter"`
}
stmt := dbq.INSERTStmt("models", []string{"name", "title", "fax", "web", "age", "\"right\"", "counter"}, 1, dbq.PostgreSQL)If you are going to query into the model (which I suspect you will), then:
Just do this instead when inserting:
stmt := dbq.INSERTStmt("models", []string{"name", "title", "fax", "web", "age", "\"right\"", "counter"}, 1, dbq.PostgreSQL)
_, err := dbq.E(context.Background(), dbqdb, stmt, nil, dbq.Struct(m)[1:] )The second way is probably what you actually want.
Now it works, but to my surprise it is much slower than go-pg and gorm.
Where am I doing wrong?
200 times - Insert
gorm: 0.85s 4249336 ns/op 7006 B/op 90 allocs/op
pg: 0.85s 4254043 ns/op 2077 B/op 11 allocs/op
dbq: 1.20s 5991529 ns/op 1857 B/op 60 allocs/op
You can try it with:
git clone https://github.com/frederikhors/orm-benchmark.git
cd orm-benchmark
git checkout add-dbq
docker-compose up -d
go run . -orm=pg -orm=dbq -orm=gorm
That's surprising to me too. I expect it to be more or less the same as gorm. I'm not familiar with pg.
temporarily, can you try directly inserting the stmt from dbq.INSERTStmt instead of using that function.
And also directly inserting the args instead of dbq.Struct() to see if the issue is with dbq.E
It's interesting that mine uses significantly less memory that gorm and slightly less than pg.
temporarily, can you try directly inserting the stmt from dbq.INSERTStmt instead of using that function.
And also directly inserting the args instead of dbq.Struct() to see if the issue is with dbq.E
I do not understand what to do.
well dbq.E is just a simple wrapper over database/sql Exec function. That's the limiting case. No package can get faster than directly using Exec function. dbq.E() is only a very light wrapper over it.
- dbq.INSERTStmt will return a string. Instead of using dbq.INSERTStmt, just use the string directly.
- replace dbq.Struct() with :
"Orm Benchmark", "Just a Benchmark for fun", "99909990", "http://blog.milkpod29.me", 100, true, 1000
If that is still slower, then there is something very seriously wrong. Something mysterious.
Done:
stmt := "INSERT INTO models ( name,title,fax,web,age,\"right\",counter ) VALUES ($1,$2,$3,$4,$5,$6,$7)"
_, err := dbq.E(context.Background(), dbqdb, stmt, nil, "Orm Benchmark", "Just a Benchmark for fun", "99909990", "http://blog.milkpod29.me", 100, true, 1000)200 times - Insert
pg: 0.83s 4126598 ns/op 2735 B/op 11 allocs/op
raw: 1.00s 5005248 ns/op 1335 B/op 20 allocs/op
dbq: 1.01s 5074871 ns/op 920 B/op 16 allocs/op
Very strange.
I used raw too: https://github.com/frederikhors/orm-benchmark/blob/add-dbq/benchs/raw.go.
For raw and dbq you are using: https://github.com/lib/pq (which is not as performant): https://github.com/frederikhors/orm-benchmark/blob/add-dbq/main.go#L13
gorm2 is using this driver: https://github.com/jackc/pgx (superior performance)
go-pg is using their own custom driver which seems to be superior to github.com/lib/pq.
My suggestion is for dbq and raw, switch to https://github.com/jackc/pgx/tree/master/stdlib (it is a database/sql adapter for github.com/jackc/pgx: https://github.com/jackc/pgx#compatibility-with-databasesql
sql.Open("pgx", "user=postgres password=secret host=localhost port=5432 database=pgx_test sslmode=disable")
See: https://github.com/jackc/pgx#comparison-with-alternatives
That will make your benchmark fairer.
Here we are with pgx, wow!
2000 times - Insert
raw: 8.17s 4084892 ns/op 750 B/op 13 allocs/op
dbq: 8.23s 4117260 ns/op 1711 B/op 54 allocs/op
pg: 8.44s 4218307 ns/op 1015 B/op 10 allocs/op
gorm: 8.47s 4233492 ns/op 6732 B/op 90 allocs/op
Now I have to complete the other tests.
stmt := dbq.INSERTStmt("models", []string{"name", "title", "fax", "web", "age", "\"right\"", "counter"}, 1, dbq.PostgreSQL)
I will make a new helper function to make extracting []string{"name", "title", "fax", "web", "age", "\"right\"", "counter"} this from the model directly.
so you can then do: dbq.StructFields(m) => return []string{...}
This dbq.Struct(m)[1:] is the other strange code we need to have. Do we need?
That is only for inserting. It is just to remove the id field (which is first value). When you are inserting, you aren't including id because its set to auto_increment/serial.
I'll add a new optional param to it so you can chose to exclude a value:
like this dbq.Struct(m, "id")
I'm trying multinsert now with this code:
func DbqInsertMulti(b *B) {
var ms []*Model
wrapExecute(b, func() {
initDB()
ms = make([]*Model, 0, 100)
for i := 0; i < 100; i++ {
ms = append(ms, NewModel())
}
})
for i := 0; i < b.N; i++ {
//for _, m := range ms {
// m.Id = 0
//}
stmt := dbq.INSERTStmt("models", []string{"name", "title", "fax", "web", "age", "\"right\"", "counter"}, len(ms), dbq.PostgreSQL)
_, err := dbq.E(context.Background(), dbqdb, stmt, nil, ms)
if err != nil {
fmt.Println(err)
b.FailNow()
}
}
}but the error is: expected 700 arguments, got 100.
Why?
Interesting. I'll add functionality to make your code work as is. it's a simple change.
In the mean time:
var ms []interface{}
ms = append(ms, dbq.Struct(NewModel())[1:]...)Depending on what you are trying to benchmark, you can move stmt = ... outside of for loop
I think I have no more time for this test.
I had set the deadline for October 29. I realized that the differences between the ORMs are minimal and that dbq has a great potential which however does not affect the comfort / speed / ram / allocations ratio of go-pg.
Will you agree with me?
Do you want to continue with this test?
dbq isn't really a ORM. It just a small layer over database/sql which I believe is all you need (provided you know what you are doing). I'm no longer a fan of ORM. I was when I used PHP/Laravel. Then I realised that the value of ORM's are actually not that great because they don't provide an enormous amount of benefits in terms of convenience.
I don't have time either for tests. It's not really important either. The reality is that Go is so fast that it doesn't matter what you use for standard web applications.
I hope you learnt about the different drivers for your exercise. It can make a difference.
Thanks for showing me some speed bumps of dbq in terms of user-friendliness. When I get some time, I'll add some helper functions in to make life slightly easier for developers.
I agree.
But I have only one thing to clarify: I would gladly go without an ORM if I could find a significantly faster alternative.
In this case go-pg is both fast and comfortable even for much more complex queries!
I regularly deal with super complex queries that ORM's can't handle. Some ORMS provide an "escape hatch" for executing complex queries, which then makes it just easier to not use a ORM.
For basic insert and select with a model, then ORM's provide some convenience only in terms of not making errors in writing the query. I find that for basic inserts and selects, you can write the query manually very easily. You can see that philosophy in dbq.
Keep working on it because maybe I'll make the next project with dbq! 🤣