ofirbarak / deilephila

A Go service that generically synchronizes (different) databases

Deilephila

A go service that sync MongoDB to Neo4j (but can easily adapt to other databases).

How it Works

Deilephila implements the infrastructure for online sync databases fast, without assuming anything about the databases' scheme or the mapping function.
Each driver supplies basic functionality such as Init and Read.
In the beginning we start listening to events from the src driver. Upon a new event, we send it to one of the parser workers, this worker responsible to handle this event.

In order to run the application write your custom configurations in configTemplate.yml. Then, write your custom Map function that receives a src event type and handle it.

| Src DB | --> | listening to events | -> | parser workers | -> | Dst DB |

Assumption: Order of events does not matter. We prefer performance over consistency, also note you can handle this in your transform function.

Quick start guide

1. Set configs following the template: configTemplate.yml
2. Write your custom callback: just implement the function Map in plugin/pluginTemplate.go
3. Pull docker image: docker pull ofirbirka/deilephila
4. Run: docker run -v /local/path/to/config:/etc/config -v /local/path/to/plugin.go:/usr/src/deilephila/plugin/pluginTemplate.go --network=host deilephila

Tests

See here

Contributing

Add a new driver

1. Implement a new driver class under drivers folder that implement the Driver interface
2. Add a test driver in test/drivers/ and tests under test/integration
3. Compile and run tests

Push new image

docker build -t deilephila .
docker tag deilephila ofirbirka/deilephila:v1
docker push ofirbirka/deilephila:v1