Project MOVED
OrientDB ETL has been moved to OrientDB core modules: https://github.com/orientechnologies/orientdb/tree/develop/etl
ETL
The OrientDB-ETL module is an amazing tool to move data from and to OrientDB by executing an ETL process. It's super easy to use. OrientDB ETL is based on the following principles:
- one configuration file in JSON format
- one Extractor is allowed to extract data from a source
- one Loader is allowed to load data to a destination
- multiple Transformers that transform data in pipeline. They receive something in input, do something, return something as output that will be processed as input by the next component
How ETL works
EXTRACTOR => TRANSFORMERS[] => LOADER
Example of a process that extract from a CSV file, apply some change, lookup if the record has already been created and then store the record as document against OrientDB database:
+-----------+-----------------------+-----------+
| | PIPELINE |
+ EXTRACTOR +-----------------------+-----------+
| | TRANSFORMERS | LOADER |
+-----------+-----------------------+-----------+
| FILE ==> CSV->FIELD->MERGE ==> OrientDB |
+-----------+-----------------------+-----------+
The pipeline, made of transformation and loading phases, can run in parallel by setting the configuration {"parallel":true}.
## Installation Starting from OrientDB v2.0 the ETL module will be distributed in bundle with the official release. If you want to use it, then follow these steps:
- Clone the repository on your computer, by executing:
git clone https://github.com/orientechnologies/orientdb-etl.git- Compile the module, by executing:
mvn clean install- Copy
script/oetl.sh(or .bat under Windows) to $ORIENTDB_HOME/bin - Copy
target/orientdb-etl-2.0-SNAPSHOT.jarto $ORIENTDB_HOME/lib
Usage
$ cd $ORIENTDB_HOME/bin
$ ./oetl.sh config-dbpedia.json
## Available Components
Examples:
Look to the Documentation for more information.