Still active?
delaneyj opened this issue · comments
The idea of using an immutable graph store in conjunction with event sourcing make a lot of sense for an upcoming project, however it looks like BadWolf has been abandoned. Is this the case?
Nop. Not sure where we you got the idea :) We are still working on it for the long run. @delaneyj what do you have in mind?
Well the last commit was months ago and no back ends are supported yet so... :)
The system would be at a high level a CMS for legal/health documents. With CQRS/event sourcing you would normally have an event log and a infrastructure around re-hydrating the state. The idea of having an immutable graph of the relationships actually makes more logical sense to me if possible. Had already started looking at doing a combination of (hexastore)[http://karras.rutgers.edu/hexastore.pdf] with an event log.
Fundamentally temporal & immutable seems like a necessity and this project seems to already have a huge head start on that.
Yes, there are not going to be drivers released in this project since that is not its goal. You can find some driver work elsewhere (see the bottom of http://google.github.io/badwolf/) :)
Yes, usually a way I have seen similar constructs are around monitoring some event log to filter and write new facts. Also, with the proper storage.Store wrapper you can also implement triggers for further down processing.
This trigger part is something on the top of the list for the next round of work, although it is there with BQL FR like filters and having condition improvements.
@delaneyj there has been some movement on actually releasing some drivers that we have built.
They are being release in the parallel badwolf-drivers repository. Currently, the main stable ones:
- Volatile: A volatile memory-based one.
- BoltDB: A persistent driver build on top of BoltDB.
There is also a branch that has a functional one based on GCP BigTable. We will merge it into the main maste brunch once we have finished ironing out some minor issues needed. But as of now, it should be good enough to take out for a spin :)
Thank you @xllora , just saw this. Will look through the bolt driver, thanks for releasing it!
Can I get some insight into the Triggers mentioned above. That would be very interesting perhaps for implementing live queries through a message queue like NATs and Google pub sub.
Anyway any insight would be useful to help me roadmap and architect Integration.
@gedw99 the idea is very simple. If you check storage.Store and storage.Graph are just interfaces. All BQL is built around them.
You can easily create a wrapper, for instance, for storage.Store. The implementation New method just accepts also storage.Store. Something like
func New(ctx context.Context, st storage.Store) storage.Store { ... }
The implementation of the interface, let's say Wrapper, just delegates the call to the wrapped implementation. Something along this lines.
type Wrapper struct {
st storage.Store
}
func New(ctx context.Context, st storage.Store) *Wrapper {
return &Wrapper{st: st}
}
func (w *Wrapper) Name(ctx context.Context) string {
return w.st.Name(ctx)
}
func (w *Wrapper) Version(ctx context.Context) string {
return w.st.Version(ctx)
}
func (w *Wrapper) NewGraph(ctx context.Context, id string) (Graph, error) {
return w.st.NewGraph(ctx, id)
}
func (w *Wrapper) Graph(ctx context.Context, id string) (Graph, error) {
return w.st.Graph(ctx, id)
}
func (w *Wrapper) DeleteGraph(ctx context.Context, id string) error {
return w.st.DeleteGraph(ctx, id)
}
func (w *Wrapper) GraphNames(ctx context.Context, names chan<- string) error {
return w.st.GraphNames(ctx, names)
}
Now you have everything to add triggers to the operations that change state for instance. Let's assume publisher.Publisher is what would allow you publish to NATS/Google Cloud PubSub. Let's say something along these lines below.
type Wrapper struct {
st storage.Store
pub publisher.Publisher
}
func New(ctx context.Context, st storage.Store, pub publisher.Publisher) *Wrapper {
return &Wrapper{
st: st,
pub: pub,
}
}
func (w *Wrapper) Name(ctx context.Context) string {
return w.st.Name(ctx)
}
func (w *Wrapper) Version(ctx context.Context) string {
return w.st.Version(ctx)
}
func (w *Wrapper) NewGraph(ctx context.Context, id string) (Graph, error) {
if err:= w.st.NewGraph(ctx, id); err != nil {
return err
}
// And you can publish it async to avoid latency increase of the op.
// If you need to guarantee publishing you will need more work here.
go func() {
w.pub.Publish(ctx, fmt.Sprintf("new grapd: %d", id))
}()
return nil
}
func (w *Wrapper) Graph(ctx context.Context, id string) (Graph, error) {
return w.st.Graph(ctx, id)
}
func (w *Wrapper) DeleteGraph(ctx context.Context, id string) error {
if err:= w.st.DeleteGraph(ctx, id); err != nil {
return err
}
// And you can publish it async to avoid latency increase of the op.
// If you need to guarantee publishing you will need more work here.
go func() {
w.pub.Publish(ctx, fmt.Sprintf("new grapd: %d", id))
}()
return nil
}
func (w *Wrapper) GraphNames(ctx context.Context, names chan<- string) error {
return w.st.GraphNames(ctx, names)
}
Now when you create a new storage driver and properly wrap it. That would look something like this for instance if you would like to add that to the bw tool
// Registers the available drivers.
func registerDrivers() {
registeredDrivers = map[string]common.StoreGenerator{
// Memory only storage driver.
"VOLATILE": func() (storage.Store, error) {
return memory.NewStore(), nil
},
"BWBOLT": func() (storage.Store, error) {
s, bdb, err := bwbolt.New(*boltDBPath, literal.DefaultBuilder(), *boldTimeout, *boltReadOnly)
db = bdb
return s, err
},
"WRAPPED_BWBOLT": func() (storage.Store, error) {
s, bdb, err := bwbolt.New(*boltDBPath, literal.DefaultBuilder(), *boldTimeout, *boltReadOnly)
if err != nil {
return nil, err
}
db = wrapper.New(context.Background(), bdb, publisher.New())
return s, err
},
}
}
thanks @xllora that looks doable. I will get going on it.