irobertson / gradle-oozie-plugin

Groovy DSL for Apache oozie workflow

Introduction

gradle-oozie-plugin provides a simple Groovy DSL for Apache oozie workflow because creating flows in XML causes serious brian damage!

Installation

maven repo: http://repository-uncommon-configuration.forge.cloudbees.com/release/
groupId: org.github.mansur.oozie
artifactId: gradle-oozie-plugin
version: 0.1

Supported workflow actions

Following workflow actions are supported

1. java
2. mapreduce
3. pig
4. hive
5. ssh
6. fs
7. shell
8. sub-workflow

In addition following decision nodes are also supported

1. start
2. end
3. kill
4. decision
5. fork and join

Usage

buildscript {
    repositories {
        maven {
            url "http://repository-uncommon-configuration.forge.cloudbees.com/release/"
        }
        mavenCentral()
    }

    dependencies {
        classpath 'org.github.mansur.oozie:gradle-oozie-plugin:0.1'

    }

}
apply plugin: 'oozie'

This is how a mapreduce flow looks like in XML

<action name='first_map_reduce'>
    <map-reduce>
      <job-tracker>http://jobtracker</job-tracker>
      <name-node>http://namenode</name-node>
      <prepare>
        <delete path='http://jobtracker/pattern' />
      </prepare>
      <job-xml>job.xml</job-xml>
      <configuration>
        <property>
          <name>mapred.map.output.compress</name>
          <value>false</value>
        </property>
        <property>
          <name>mapred.job.queue.name</name>
          <value>queuename</value>
        </property>
      </configuration>
    </map-reduce>
    <ok to='end' />
    <error to='fail' />
  </action>

and this is the same flow in groovy dsl

oozie {
first_map_reduce = [
            name: "first_map_reduce",
            type: "mapreduce",
            delete: ["${jobTracker}/pattern"],
            jobXML: "job.xml",
            ok: "end",
            error: "fail",
            configuration: [
                    "mapred.map.output.compress": "false",
                    "mapred.job.queue.name": "queuename"
            ]
    ]
}

see build.gradle and generated workflow.xml for a complete workflow