gpotts / nifi-scripting

NiFi Dynamic Script Executors

NiFi-Scripting

NiFi Dynamic Script Executors

Work-in-Progress

The goal of this project is to enable processing NiFi FlowFiles using scripting languages.

1. ExecuteJavaScript Execute supplied javaScript with arguments configured. Use case: JSON -> Mapping -> JSON
2. ExecuteGroovy: Execute supplied groovy script with arguments configured.
3. ExecuteGroovyLdap: Provide configured LDAP object to the script. Execute supplied groovy script with arguments configured. See Groovy LDAP
4. ExecuteRemoteProcess: Similar to NiFi built-in ExecuteProcess but run on remote host. See Groovy SSH

Install NiFi

1. Manual: Download Apache NiFi binaries and unpack to a folder.
2. On Mac: brew install nifi

Deploy NAR files.

# Assume you unpacked nifi-0.5.1-bin.zip to /Developer/Applications/nifi
./gradlew clean deploy -Pnifi_home=/Developer/Applications/nifi

On Mac

gradle clean deploy -Pnifi_home=/usr/local/Cellar/nifi/0.5.1/libexec

Start NiFi

cd /Developer/Applications/nifi
./bin/nifi.sh  start
./bin/nifi.sh  stop

On Mac

# nifi start|stop|run|restart|status|dump|install
nifi start 
nifi status  
nifi stop 
# Working Directory: /usr/local/Cellar/nifi/0.5.1/libexec

Testing

Upload the sample flow into NiFi and use test data and below javascript for testing:

function convert(val) {
    var g = JSON.parse(val);
    var d = {
        widget: g.widget.window.title,
        imageURI: g.widget.image.src
    };
    return JSON.stringify(d);
}

print("Hello from inside scripting!");

var  fs = util.flowFileToString(flowFile, session);
log.error(fs);
var flowString = convert(fs);
log.error(flowString);

//flowFile = session.importFrom(buff, true, flowFile);
session.remove(flowFile);
flowFile = util.stringToFlowFile(flowString, session);


flowFile = session.putAttribute(flowFile, "JS", 2222 );

ExecuteRemoteProcess testing

SSH Config DSL

remotes {
    web01 {
        role 'masterNode'
        host = '192.168.1.5'
        user = 'sumo'
        password = 'demo'
        knownHosts = allowAnyHosts
    }
    web02 {
        host = '192.168.1.5'
        user = 'sumo'
        knownHosts = allowAnyHosts
    }
}

Run DSL

ssh.run {
    session(ssh.remotes.web01) {
          result = execute 'uname -a' 
    }
}

Build

gradle nar

If you are using MapR hadoop distribution

1. Follow steps NiFi Hadoop Library for MapR

2. Set auth login config in $NIFI_HOME/conf/bootstrap.conf

   java.arg.15=-Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf

TODO

1. Support adding popular javaScript libraries (lodash.js, moment.js etc.,) via processor configuration.
2. ExecuteRemoteProcess: add expression language support for RUN_DSL.

Reference

1. Groovy Script
2. java8-nashorn-tutorial
3. Mapping Complex JSON Structures With JDK8 Nashorn
4. Groovy SSH
5. See document of Gradle SSH Plugin for details of DSL.
6. Groovy Goodness: Store Closures in Script Binding