Roadmap for v1
a8m opened this issue · comments
List of features/changes we want to finish before releasing an official v1 version:
- PostgreSQL support - done
- Eager loading - #91 - done
- User defined IDs - #127 - done
- Hooks - #377 - done
- Bulk insert - done #236
- OSS GraphQL integration - done (expected more work in the near future)
- Privacy layer - done (see entgo.io/docs/privacy)
- User defined types - WIP #277
- Custom Go Type
- Custom Dialect Type
- "Other" type
- Custom predicates (WIP)
- JSON operations
- Predicates and filters
-
Appendsupport
- Upsert and batch upsert (dialect specific)
- Query hooks (interceptors) - #833
- Better migration
- offline and versioned migration
- edge migration ({O2O, O2M} -> M2M).
- Multi-level caching (Experimental solution ariga/entcache)
- Cluster support (read replicas)
- Horizontal sharding
- Edge fields (metadata on edges)
- Cascading deletion - #407
- DDL support
- Application-level support
Please note that this list is changing.
Can you elaborate on Cluster support (read replicas).
Sounds like queries go to the Read replicas and mutations to a master.
In which case you will need a CDC process ?
Eager to know this.
Sounds like queries go to the Read replicas and mutations to a master.
Exactly. We want to add a cluster-client option (leader and 1 or more read replicas).
In which case you will need a CDC process ?
Can you elaborate, I'm not sure I got your question.
CDC is short for Change Data Capture.
Implementations often use binlog parsing of the Master ( read write db), and detect the db changes and then pump them out over a stream so the Slave ( Read only DB) can update its data.
Here is a great example with cockroachdb.
https://www.cockroachlabs.com/docs/stable/change-data-capture.html
for MySQL thee is this: https://github.com/uber/storagetapper
for PostreSQLthere is this: https://github.com/samsarahq/thunder/blob/v0.5.0/livesql/binlog.go
I have a request for the Roadmap.
A change notification so that if a record changes the system tells me. Its basically CDC.
For me the use case is to make it easy to have Read only VIew models that might be a different structure from the READ / Write models. Sort of like what you do with CQRS.
For me i dont need any of the network code because i might want to use a different netowrk protocol.
But it woudl be good to describe the change feed using a Protobuf because Protobufs support Schema Evolution which is really important and often forgotten about. Also protobuf makes it easy to integreate other languges.
Do you have a translation plan for other languages ?
@winwisely99 I think the use of read replicas and clustering here means, that ent will support providing separate master and read connection details, but the actual replication should be done by the database cluster.
CQRS like systems usually build on domain operations, thus a change is actually detected by the domain layer, not the storage layer. If you want to detect changes, you can set up triggers or check the return value of the MySQL DML query. I hardly think this should be the job of an ORM, because it's either above or below it.
But that's my two cents only.
Will there be support for GraphQL? I am working with a graph database that does not support Gremlin but rather GraphQL which means only an underlying driver would have to be written if I understand it correctly.
Do you have a translation plan for other languages ?
Not at the moment.
I think the use of read replicas and clustering here means, that ent will support providing separate master and read connection details, but the actual replication should be done by the database cluster.
Exactly.
Will there be support for GraphQL?
There are plans to add support for API layers, like GraphQL and REST (Swagger for example).
However, it's not in our roadmap to v1, because we see it as an extension to ent that can be done with an external plugin.
I am working with a graph database that does not support Gremlin but rather GraphQL which means only an underlying driver would have to be written if I understand it correctly.
About GraphQL as a storage driver - not part of v1 roadmap.
@a8m any eta for eager loading and hooks?
@a8m any eta for eager loading and hooks?
We are in the holiday season right now, and we'll get back to work on Oct 23rd (so I can't commit to an exact date). However, I can say that we plan to tackle immediately these 2 problems (eager-loading first). You are welcome to help us with the design of the eager-loading API (#91).
Right now, I'm sporadically working on adding the PostgreSQL support.
Would there be interest in support for Dgraph? We could help out if there’s interest (I’m the founder of Dgraph).
Hey @manishrjain, thanks for proposing your help.
Would there be interest in support for Dgraph?
Currently, we want to focus on our roadmap and don't want to commit to something we don't have the capacity for it (because it means maintaining and supporting a storage-driver we're not using nor have experience with).
However, @alexsn and I just discussed this morning on adding a public interface for pluggable storage-drivers. So, if someone is willing to take and maintain this, I'll be happy to help.
It would be great to have support for UUIDs as IDs with explicit handling of the ID generation (instead of auto-generating the IDs)
It would be great to have support for UUIDs as IDs with explicit handling of the ID generation (instead of auto-generating the IDs)
We do have plans to add support for custom ID generation but note that with universal IDs you're able to determine ent type by ID which you won't be able to do with UUIDs.
Does eager loading means using the practices of dataloader? If so, there's https://github.com/graph-gophers/dataloader that can be used.
As per hooks, I don't know what you have in mind but I saw a neat implementation by https://github.com/infobloxopen/protoc-gen-gorm
type EntUserWithBeforeCreate interface {
EntUserBeforeCreate(context.Context, *EntUser) error
}
...
if prehook, ok := interface{}(m).(EntUserWithBeforeCreate); ok {
if err = prehook.EntUserBeforeCreate(ctx, &user); err != nil {
return to, err
}
}Hope this helps
Does eager loading means using the practices of dataloader? If so, there's https://github.com/graph-gophers/dataloader that can be used.
dataloader can be used in combination with eager loading where you collect up a bunch of requested fields over some time period and run a single query to fetch all of those fields.
Will there be support for MSSql, Redis, Elasticsearch?
Will there be support for MSSql, Redis, Elasticsearch?
No, at least in the foreseeable future. However, as I mentioned above, we have plans to add a public interface for pluggable storage-drivers. So, if someone want to take and maintain it, I'll be happy to help.
Please add geo types and location queries to roadmap.
Any plans to add ON DUPLICATE KEY UPDATE/ON CONFLICT DO UPDATE?
No, at least in the foreseeable future. However, as I mentioned above, we have plans to add a public interface for pluggable storage-drivers. So, if someone want to take and maintain it, I'll be happy to help.
Hey @a8m, I'm interested in seeing an interface for pluggable storage drivers. Do you have much vision on what this should look like? Would it be reasonable to first build out pluggable infrastructure for SQL dialects rather than something more general? If you're not opposed I'd like to open an issue to start discussing design requirements and a path forward.
Hey @ajwerner, and sorry for the delayed response.
Hey @a8m, I'm interested in seeing an interface for pluggable storage drivers. Do you have much vision on what this should look like?
I can share with you what I thought about - A "runtime" storage driver (in the templates), which generates code that expect for an interface that wraps a list of operations, like: CreateNode, UpdateNode, etc.
We wait with this, because we want to finish first all the critical parts in the codegen (like hooks).
Would it be reasonable to first build out pluggable infrastructure for SQL dialects rather than something more general?
Maybe. I'm open for this (I guess you talk about cockroachdb integration).
If you're not opposed I'd like to open an issue to start discussing design requirements and a path forward.
Let's open an issue and discuss about it there.
Sorry for so many requests, but i am currently using your lib and intend to continue to do so :) There is few things that i lack now.
Cursor
Is there are any plans to add cursor or iterator to query API?
I mean something like
pets, err := client.Pets.Query().Iterator(ctx) // instead of All(ctx)
if err != nil {
// ...
}
for pets.Next() {
pet, err := pets.Only() // not sure about naming, maybe pets.Get(), or pets.One()
if err != nil {
// ...
}
}
If you need to load and process a lot of data from database, it is better to do this with cursor/iterator.
Alternative struct-initialized API
Is there are any plans to introduce alternative API for entity creation?
This is how to create user now:
a8m := client.User.
Create().
SetAge(30).
SetName("a8m").
AddGroups(hub, lab).
SaveX(ctx)
This is how to create user with initialized struct:
a8m := &ent.User{
Age: 30,
Name: "a8m",
Groups: []*ent.Group{
hub,
lab,
},
}
a8m = client.User.Create(a8m).SaveX(ctx)
This is not shorter, but when i writing with setter-like API, the IDE sometimes makes me crazy, advising me wrong completions. It is not problem of the lib, but maybe alternative API will be more suitable for community?
Also this maybe a step ahead to implement #236
var users = []*ent.User{ ... }
err := client.User.CreateMany(users).Save(ctx)
Any discuss appreciated.
Iterators are good, but they use a single-threaded processing style. After reading the next record, the connection to the database will be idle for the duration of processing. I implemented concurrency with a buffered channel, which will be more efficient. See #362
UPD: iterators that send to the channel sounds good!
what's the expected release date?
Hey @fr3fou - We don't have an official release date.
However, we're actively working on ent and we made good progress until now - I'm keeping the first comment updated.
We invest time in features design (more than expected) and collect feedbacks from our users in order to provide high quality product. Therefore, we try to avoid working under pressure (as much as possible) and prefer to not commit to a specific date.
does ent support/plan to support seeding database? it currently does migration so wouldn't want to seed using another tool
Seeding would be an absolute killer feature
Just to clarify, does Edge fields (metadata on edges) mean adding additional columns to edges?
What we're looking for is to be able to say something similar to:
user -> likes/follows/blocked -> user
using edges. This would mean being able to define more than just 1-1 relationships, but quantifying the relationship.
appreciate an insane amount of work might be going on behind the scenes on this project but it'll be good if some regular updates can be shared?
i'm currently blocked from using this in production waiting for User defined types and Cascading deletion :(
Any updates appreciated! @a8m
Hey @moh-abk,
Please note that the first comment (with the task list) is continuously updating with the status, and also, the public releases contain information about the changes that were added to ent. See the latest release.
Is there any planned support for raw SQL queries?
Edit: Just saw #85, sorry.
I love this package!
You provide the Timestampable-Mixin, any chance to add some kind of Blameable-Mixin?
I love this package!
You provide the Timestampable-Mixin, any chance to add some kind of Blameable-Mixin?
Not sure what that mean but feel free to submit a PR.
I love this package!
You provide the Timestampable-Mixin, any chance to add some kind of Blameable-Mixin?Not sure what that mean but feel free to submit a PR.
I would, but i do not know how to implement what i mean.
I often have 2 fields attached to my entities: updated_by and created_by. Whom to blame for creating and updating this entity so to speak.
But since the values of those fields are no simple go types but another entity, it should be an edge. And i do not know how to abstract that for all possible cases.
I am happy to help, if you can point me in one direction.
@masseelch I created #830 to talk about this further. I am also interested in implementing a general auditing hook
Hey @masseelch, let's move this discussion to #830. Thanks @cliedeman for creating this issue 😃
What's you opinion on adding #400 to roadmap for v1? It's the only feature I am currently missing in ent. How difficult is it to implement?
Is there any plans for query caching to be built in (say to redis or memcached)?
Is there any plans for query caching to be built in (say to redis or memcached)?
Yes, but it's currently blocked on #833.
Is there any plans for query caching to be built in (say to redis or memcached)?
Yes, but it's currently blocked on #833.
what's the expected release date?
Seeing as the privacy layer is marked as complete, can we expect a release that contains this functionality soon?
Thanks a lot for your hard work on this, love the package!
Thanks for the feedback @BrentChesny, it is great to hear that.
I will release a new version in the upcoming days, but if it's urgent, you can just use the latest master (it's stable).
go get github.com/facebook/ent@master
Ex-Facebook employee here, wondering are there plans to have Ent support in other language as well? Would this open source Ent have almost all the functionality as the internal Ent?
Ex-Facebook employee here, wondering are there plans to have Ent support in other language as well? Would this open source Ent have almost all the functionality as the internal Ent?
you could just use the GRPC that is codegenerated ?
@winwisely99 I think the use of read replicas and clustering here means, that ent will support providing separate master and read connection details, but the actual replication should be done by the database cluster.
CQRS like systems usually build on domain operations, thus a change is actually detected by the domain layer, not the storage layer. If you want to detect changes, you can set up triggers or check the return value of the MySQL DML query. I hardly think this should be the job of an ORM, because it's either above or below it.
But that's my two cents only.
I know, but i prefer a pure SQL level of CQRS.
Like this:
https://materialize.com/
https://materialize.com/docs/sql/create-source/
so then your can migrate easily and then gen your Protobufs of the Ent Schema
Curious if Ent's OSS GraphQL roadmap includes Apollo Federation support (being federat_ed_). We are using Federation to compose multiple GraphQL services into one graph. Presently, we can use federated DGraph for a single service, but we also have some Federated Go services that are really just adaptors for relational databases (PostgreSQL). We haven't been able to adopt Ent and avoid the adapter service because we needed Apollo Federation support. Also curious if any Ent / DGraph combination has been tried or considered.
We haven't been able to adopt Ent and avoid the adapter service because we needed Apollo Federation support.
Hi, Ent use gqlgen to integration GraphQL feature, they also support Apollo Federation.
@a8m Any chance for SQL Server support?
Hi Ariel, we have been using the ent framework successfully, you did a great job, many thanks for that. we have a dependency on Edge fields (metadata on edges). when can we expect this change?
thanks
Agnel
Hey @agnel-foyernet, you can follow this feature on #1949
We're almost done with the new migration. MySQL and PostgreSQL are ready, and SQLite and MariaDB are in progress. Once it's done, we'll post a proper roadmap update with some of the most requested features in the issue tracker (new GraphQL integration, polymorphism, metadata on edges, and much more).
Logging support like https://gorm.io/docs/logger.html
:-O Wow, that was totally unexpected! Great news! 🚀 🚀 🚀
Fly.io have cluster support that is a generic pattern for databases.
https://fly.io/docs/reference/postgres/
they detect a write and then replay the request onto the writable db.
This pattern can be used for any database.
For replication they use stolen. That’s not database agnostic.
Congrats for the new release!
I was looking at the next items on the roadmap and noticed that "Cluster support (read replicas)" is not there. Any ideas when this could become a thing?
For large (even mid) scale applications this is a must and it doesn't feel like to be a very complicated thing (I could be absolutely wrong though). In other words: this could be a low hanging fruit. The alternative is using various proxy solutions, but I'm not sold on them just yet.
I don’t know if there is a design for it yet .
If you have a suggestion , put it forward
I think it was one of the first questions in this thread and was quickly clarified what it means. Don't know if any further details in terms of design help deciding it's place on the roadmap. I can certainly come up with some ideas for the interface if that helps.
I don't know either. I liked the way fly.io solved their db scaling issue in a very pragmatic way.
https://fly.io/docs/getting-started/multi-region-databases/
https://www.youtube.com/watch?v=IqnZnFpxLjI
- jump to 12 minutes in...
Congrats for the new release!
Thanks @sagikazarmark
I was looking at the next items on the roadmap and noticed that "Cluster support (read replicas)" is not there. Any ideas when this could become a thing?
There is an example of how to implement a "cluster driver" in #1580 (comment). I also added it to our website.
I still think we should make it an official plugin/extension with additional features we created internally. We just didn't get to it yet.
@a8m i had no idea this was done. both design and code.
Its not "ready" yet ?
Is there a repo anywhere that exercises this, so we can play around with it. ?
If feels like the Cache is where this "redirection" would be between primary ( write only) and secondary ( read only ) db's ?
https://github.com/ariga/entcache/blob/master/README.md#high-level-design
Thanks for the links @a8m! I'll take a look at them.
Add support for cockroachdb.
Hello @StarpTech, someone on discord said that they are using CockroachDB without problems with Ent.
Feel free to join the Ent Discord https://discord.gg/qZmPgTE6RX
Hi @crossworth the difference between "it works" and support is huge. I'd like to see cockroachdb as part of the CI tests.
Please add support for Joins. The eager load mechanism is too limited.
Currently, it's not possible to filter on the dataset of an entity with multiple eager loaded edges. The filter capabilities are scoped to one specific edge at a time.
Imagine you would like to get all authors where posts OR books starts with the letter "a". Each author has two edges (1. Posts, 2. Books) We can't query authors with either a matched post or book because each edge is loaded in a separate database query.
This is a very fundamental feature. I'm curious what's the strategy of ent here? Did I forget something? 🤔
Updated: It is possible through the .hasX methods in the .Where clause of the starting node https://entgo.io/docs/traversals
Do you have a plan for Horizontal sharding ?
Do you have a plan for Horizontal sharding ?
Hi, just wanted to say, Ent is fantastic! The quality speaks for itself in the shear volume of developers using it in production.
Therefore is super strange, and inconvenient, that ent does not have a 1.0 version.
Looking at the above list. These look like 1.1 or 2.0 features. We could easily have a 1.0 version today. All you have to do is change that number and send out a release. The problem with 0.11 is that 0.12 can be a breaking change so we all have to be super, extra careful when upgrading even if all we want is that small bug fix.
Compared with new features, it's more meaningful to facilitate the usage and improve frequently used functionality.