nifi-formula
A salt formula to deploy Apache NiFi
Configure Apache NiFi in 3 different configurations:
- Standalone
- Cluster with embedded Zookeeper
- Cluster with external Zookeeper
Configures systemd and sysctl limits automatically. See the pillar.example for detailed list of pillars.
Table of Contents
General notes
See the full SaltStack Formulas installation and usage instructions.
If you are interested in writing or contributing to formulas, please pay attention to the Writing Formula Section.
If you want to use this formula, please pay attention to the FORMULA file and/or git tag,
which contains the currently released version. This formula is versioned according to Semantic Versioning.
See Formula Versioning Section for more details.
If you need (non-default) configuration, please pay attention to the pillar.example file and/or Special notes section.
Contributing to this repo
Commit message formatting is significant!!
Please see How to contribute for more details.
Special notes
Requirements
- CentOS, Ubuntu, Debian
- Cluster configurations require hosts setup with FQDN and DNS.
Sample Pillars
Pillars - Standalone Node
By Default all of the settings will configure a standalone NiFi Node
nifi:
pkg:
name: nifi
downloadurl: https://mirror.csclub.uwaterloo.ca/apache/nifi/1.12.1/nifi-1.12.1-bin.tar.gz
version: 1.12.1
installdir: /opt
# JDK Package to install. Leave empty to not install.
javajdk: java-1.8.0-openjdk
nifi:
cluster.is.node: 'false'
systemdconfig:
user: root
group: root
limitnofile: 50000
limitnproc: 10000
Pillars - 3 Node Cluster with Embedded Zookeeper
nifi:
pkg:
name: nifi
downloadurl: https://mirror.csclub.uwaterloo.ca/apache/nifi/1.12.1/nifi-1.12.1-bin.tar.gz
version: 1.12.1
installdir: /opt
# JDK Package to install. Leave empty to not install.
javajdk: java-1.8.0-openjdk
systemdconfig:
user: root
group: root
limitnofile: 50000
limitnproc: 10000
nifi:
# cluster node properties (only configure for cluster nodes) #
cluster.is.node: 'true'
cluster.node.address: {{ grains['fqdn'] }}
cluster.node.protocol.port: '1111'
cluster.flow.election.max.wait.time: '1 mins'
zookeeper.connect.string: 'nifi-1.localdomain:2181,nifi-2.localdomain:2181,nifi-3.localdomain:2181'
state.management.embedded.zookeeper.start: 'true'
web.http.host: {{ grains['fqdn'] }}
zookeeperproperties:
# To configure Zookeeper.properties set 'state.management.embedded.zookeeper.start' to 'true' above, and then define your Embedded Zookeeper servers here.
customservers:
Node1:
hostname: nifi-1.localdomain
zookeeper_myid: 1
zookeeper_clientPort: 2181
zookeeper_peerPorts: '2888:3888'
Node2:
hostname: nifi-2.localdomain
zookeeper_myid: 2
zookeeper_clientPort: 2181
zookeeper_peerPorts: '2888:3888'
Node3:
hostname: nifi-3.localdomain
zookeeper_myid: 3
zookeeper_clientPort: 2181
zookeeper_peerPorts: '2888:3888'
Pillars - 3 Node Cluster with External Zookeeper Servers
nifi:
pkg:
name: nifi
downloadurl: https://mirror.csclub.uwaterloo.ca/apache/nifi/1.12.1/nifi-1.12.1-bin.tar.gz
version: 1.12.1
installdir: /opt
# JDK Package to install. Leave empty to not install.
javajdk: java-1.8.0-openjdk
systemdconfig:
user: root
group: root
limitnofile: 50000
limitnproc: 10000
nifi:
# cluster node properties (only configure for cluster nodes) #
cluster.is.node: 'true'
cluster.node.address: {{ grains['fqdn'] }}
cluster.node.protocol.port: '1111'
cluster.flow.election.max.wait.time: '1 mins'
zookeeper.connect.string: 'nifi-1.localdomain:2181,nifi-2.localdomain:2181,nifi-3.localdomain:2181'
state.management.embedded.zookeeper.start: 'true'
web.http.host: {{ grains['fqdn'] }}
Available states
nifi
Meta-state (This is a state that includes other states).
This installs the nifi package, manages the nifi configuration file and then starts the associated nifi service.
nifi.package
This state will install the nifi package only. This downloads the tar.gz file from the downloadurl and deploys it to servers.
nifi.config
This state will configure the nifi service and has a dependency on nifi.install
via include list.
nifi.service
This state will start the nifi service and has a dependency on nifi.config
via include list.
Testing
Linux testing is done with kitchen-salt.
Requirements
- Ruby
- Docker
$ gem install bundler
$ bundle install
$ bin/kitchen test [platform]Where [platform] is the platform name defined in kitchen.yml,
e.g. debian-9-2019-2-py3.
bin/kitchen converge
Creates the docker instance and runs the nifi main state, ready for testing.
bin/kitchen verify
Runs the inspec tests on the actual instance.
bin/kitchen destroy
Removes the docker instance.
bin/kitchen test
Runs all of the stages above in one go: i.e. destroy + converge + verify + destroy.
bin/kitchen login
Gives you SSH access to the instance for manual testing.