Implement `sql.Scanner` and `driver.Valuer` for `timestamppb.Timestamp` in protobuf-go
paulheg opened this issue · comments
Is your feature request related to a problem? Please describe.
I want to use the structs generated by protoc to be stored in a database.
This is working perfectly fine until I want to use timestamps.
It seems the common datastructure for timestamps is google.protobuf.Timestamp
When using libraries like sqlx, or bun, they make use of the sql.Scanner and driver.Valuer interfaces,
to make scanning much easier.
Describe the solution you'd like
I would like, as statet above, to implement sql.Scanner and driver.Valuer in timestamppb.Timestamp to convert to time.Time and vice versa,
as this is commonly used in database libraries.
Describe alternatives you've considered
I could create a custom type that is based on timestamppb.Timestamp and implement the interfaces there.
This is not really practicable as I would have to copy values manually. Not elegant.
I am open and very thankful for any other ideas.
timestamppb.Timestamp has method AsTime to convert to time.Time.
That is correct, but how do I then get the database library to use this?
What I have so far:
package database
import (
"database/sql"
"database/sql/driver"
"errors"
"time"
"google.golang.org/protobuf/types/known/timestamppb"
)
var _ sql.Scanner = &CustomTimestamp{}
var _ driver.Valuer = &CustomTimestamp{}
type CustomTimestamp timestamppb.Timestamp
// Value implements driver.Valuer.
func (t CustomTimestamp) Value() (driver.Value, error) {
a := timestamppb.Timestamp(t)
return a.AsTime(), nil
}
// Scan implements sql.Scanner.
func (t *CustomTimestamp) Scan(src any) error {
var source timestamppb.Timestamp
switch src.(type) {
case time.Time:
source = *timestamppb.New(src.(time.Time))
default:
return errors.New("invalid type for timestamppb.Timestamp")
}
*t = CustomTimestamp(source)
return nil
}
Not sure if it is helpful to you, but we did something similar here for GORM.
🤔 How certain could we be that this conversion in sql.Scanner and driver.Valuer would be the correct and only way to convert them?
Is simply wrapping them through time.Time sufficient, and complete?
P.S.: My fear here, is that we implement “the obvious answer” and someone who isn’t expecting that answer, or who wants a different answer is left out-of-luck, because we forced the decision?
Hi @puellanivis thank you for your comment.
Looking at the driver.Valuer interface documentation it states:
// Value is a value that drivers must be able to handle.
// It is either nil, a type handled by a database driver's NamedValueChecker
// interface, or an instance of one of these types:
//
// int64
// float64
// bool
// []byte
// string
// time.TimeThis means time.Time must be supported as it is expected.
I would say it is better to be able to store google.protobuf.Timestamp with a database driver out of the box.
If someone wants a different format, they can do it the same way I have to do it right now, by creating custom database structs and convert them manually to whatever is desired.
I dont think there is a way to make the current situation worse.
If you have alternative solutions please let me know.
👍 I’m all in then @paulheg
Just in case it helps, here is where the code lives: https://github.com/protocolbuffers/protobuf-go/tree/master/types/known/timestamppb
To send a contribution, see https://github.com/protocolbuffers/protobuf-go/blob/master/CONTRIBUTING.md for instructions.
Thank you!
I'm sorry, but this isn't a change we can make.
It is not practical for every type to implement every serialization interface. There's a good argument to be made that the encoding package interfaces are sufficiently pervasive that implementing them is practical, but sql.Scanner is not pervasive. For example, in the standard library net.IP, netip.Addr, and regexp.Regxep all implement encoding.TextUnmarshaler, but none of them implement sql.Scanner.
The driver.Valuer interface is interface { Value() (driver.Value, error) }. Implementing this interface requires importing on sql/driver. We can't add this dependency to generated messages, as it would require every program using timestamppb to import sql/driver regardless of whether it uses SQL or not.
This issue is about timestamppb.Timestamp, but Timestamp is just one message and I don't see why we would special case it. If we were to add support for database/sql to one message, we should do it for all messages.
In addition, adding this support is at most a small convenience. Converting a time.Time to and from a Timestamp is simple:
var ts time.Time
if err := rows.Scan(&ts); err != nil {
// handle error
}
p := timestamppb.New(ts)
rows, err := stmt.Query(p.AsTime())
Hi @neild,
Im sorry to hear this, unfortunately I cant really follow your argumentation.
It is not practical for every type to implement every serialization interface.
No one is asking for this.
If we were to add support for database/sql to one message, we should do it for all messages.
Why? Time is a very common datatype to store in the database. I had no problems with other messages since they consist of basic types that are supported anyway. This is literally the only roadblock.
In addition, adding this support is at most a small convenience.
Have you ever tried to scan a bigger message?
Almost every database model contains a created_at etc. timestamp field.
Hence this interface exists and tools like sqlx.
The import required to support sql.Valuer is unfortunate. And while sql.Scanner doesn’t require any such import, I wonder what its value would be without the parallel sql.Valuer implementation. :(
I ran into this same issue trying to reuse the protobuf generated structs for mapping to database result objects using sqlx.
var x protos.User // protoc generated struct with a CreatedAt field `CreatedAt *timestamppb.Timestamp`
res := s.DB.QueryRowx("SELECT * FROM user WHERE id = $1", id)
err := res.StructScan(&x)
I can understand not wanting to import sql/driver into the package. Are there any suggested alternatives?
I can't see how to get around this if I can't control the generated type and I can't extend methods on that type.
Hi @jsnb,
I dont think there is a good solution for sqlx right now.
As mentioned by @meling, there is an option to register a custom serializer in gorm.
I dont think there is something equivalent implemented in sqlx and unfortunately the project does not seem to be well maintained, so I dont think there is hope to get something like this added in the near future.
If I were you, I would create new structs for database handling and make the timestamppb conversion there and use the StructScan on that new struct.
As a last resort, @neild described how to do it all manually.
Based on the discussion, I’ll close this issue.
Larger programs often use a DAL (Database Access Layer) to interact with a SQL database. The sqlc program can be used to auto-generate these based on SQL schemas and queries.
I think it’s best to treat each layer separately. That is, use protobuf for network serialization (receive/send RPCs), but use a DAL for database serialization (store/load database entries).
We see protobuf as a serialization library to be used at the edges of a program, not as an alternative to declaring struct types that a program works with. If multiple layers are involved, the program needs to translate between them. For the timestamp type, this is done by translating to a time.Time value, which then can be passed to the DAL, possibly in a sql.NullTime container.