mathieu-rossignol / ansible-nifi

Ansible Role - NiFi

Ansible Role: NiFi

An Ansible Role that installs NiFi on Linux. By default, it installs NiFi in a way that makes upgrading painless.

Requirements

Requires at least Java 8.

Role Variables

See defaults/main.yml for all variables and how to specify them. For a deeper dive, the NiFi System Administrator’s Guide is a great resource.

The following specifies where to install NiFi, along with a home directory (which will be symbolically linked to the release). Also, a centralized config directory to store files that need not be changed (to avoid copying during upgrades).

nifi_config_dirs:
  install: /opt/nifi/releases
  home: /opt/nifi/releases/current
  external_config: /opt/nifi/config_resources
  run: {{ nifi_config_dirs.home }}/run
  logs: {{ nifi_config_dirs.home }}/logs

By default, this is the directory structure that will be created:

|--opt/
  |--nifi/
    |--releases/
      |--current -> nifi-1.9.1/
      |--nifi-1.8.0/
      |--nifi-1.9.1/
    |--config_resources/
      |--archive/
      |--authorizations.xml
      |--content_repository/
      |--custom_nars/
      |--database_repository/
      |--flow.xml.gz
      |--flowfile_repository/
      |--provenance_repository/
      |--state/
      |--users.xml

Any key/value pair from a config file can be added to the following dicts. Dict names correspond to file names. The current config options for these files can be found here.

nifi_properties:
bootstrap:
logback:
login_identity_providers:
state_management:
authorizers:
zookeeper:

Dependencies

None.

Example Playbooks

These assume you have hash_behaviour=merge set in your config. If not, please also include the default dict key/values from defaults/main.yml.

Basic single node NiFi instance:

- hosts: nifi_servers
  become: yes
  roles:
    - cavemandaveman.nifi

Basic 3 node NiFi cluster using embedded Zookeeper:

- hosts: nifi_servers
  become: yes
  roles:
    - cavemandaveman.nifi
  vars:
    nifi_properties:
      nifi.web.http.host: "{{ ansible_fqdn }}"
      nifi.web.http.port: 8080
      nifi.cluster.is.node: true
      nifi.cluster.node.address: "{{ ansible_fqdn }}"
      nifi.cluster.node.protocol.port: 11443
      nifi.cluster.flow.election.max.candidates: 3
      nifi.cluster.load.balance.host: "{{ ansible_fqdn }}"
      nifi.cluster.load.balance.port: 6342
      nifi.state.management.embedded.zookeeper.start: true
      nifi.zookeeper.connect.string: nifi_server1:2181,nifi_server2:2181,nifi_server3:2181
    state_management:
      /stateManagement/cluster-provider/property[@name="Connect String"]: "{{ nifi_properties['nifi.zookeeper.connect.string'] }}"
    # Assuming nifi_server1 = 192.168.1.10, nifi_server2 = 192.168.1.11, nifi_server3 = 192.168.1.12
    # we have Ansible automatically set the myid file on each host to last octet of the node's IP address
    # and we set the 'X' of the zookeeper['server.X'] keys to those same numbers.
    zookeeper_myid: "{{ ansible_default_ipv4.address.split('.')[-1] }}"
    zookeeper:
      server.10: nifi_server1:2888:3888
      server.11: nifi_server2:2888:3888
      server.12: nifi_server3:2888:3888

Secure single node NiFi instance with LDAP:

- hosts: nifi_servers
  become: yes
  roles:
    - cavemandaveman.nifi
  vars:
    nifi_properties:
      # HTTP properties must be unset for HTTPS to work
      nifi.web.http.host: ""
      nifi.web.http.port: ""
      nifi.web.https.host: "{{ ansible_fqdn }}"
      nifi.web.https.port: 9443
      nifi.security.keystore: /path/to/keystore.jks
      nifi.security.keystoreType: JKS
      nifi.security.keystorePasswd: keystorePassword
      nifi.security.keyPasswd: keyPassword
      nifi.security.truststore: /path/to/truststore.jks
      nifi.security.truststoreType: JKS
      nifi.security.truststorePasswd: truststorePassword
    login_identity_providers:
      /loginIdentityProviders/provider/identifier: ldap-provider
      /loginIdentityProviders/provider/property[@name="Authentication Strategy"]: SIMPLE
      /loginIdentityProviders/provider/property[@name="Manager DN"]: cn=nifi,ou=people,dc=example,dc=com
      /loginIdentityProviders/provider/property[@name="Manager Password"]: password
      /loginIdentityProviders/provider/property[@name="Url"]: ldap://hostname:port
      /loginIdentityProviders/provider/property[@name="User Search Base"]: OU=people,DC=example,DC=com
      /loginIdentityProviders/provider/property[@name="User Search Filter"]: sAMAccountName={0}
    authorizers_identifiers:
      - file-user-group-provider
      - ldap-user-group-provider
      - composite-user-group-provider
    authorizers:
      /authorizers/userGroupProvider/property[@name="Initial User Identity 1"]: cn=John Smith,ou=people,dc=example,dc=com
      /authorizers/accessPolicyProvider/property[@name="Initial Admin Identity"]: cn=John Smith,ou=people,dc=example,dc=com

Secure 3 node NiFi cluster with LDAP using embedded zookeeper:

- hosts: nifi_servers
  become: yes
  roles:
    - cavemandaveman.nifi
  vars:
    nifi_properties:
      # HTTP properties must be unset for HTTPS to work
      nifi.web.http.host: ""
      nifi.web.http.port: ""
      nifi.web.https.host: "{{ ansible_fqdn }}"
      nifi.web.https.port: 9443
      nifi.security.keystore: /path/to/keystore.jks
      nifi.security.keystoreType: JKS
      nifi.security.keystorePasswd: keystorePassword
      nifi.security.keyPasswd: keyPassword
      nifi.security.truststore: /path/to/truststore.jks
      nifi.security.truststoreType: JKS
      nifi.security.truststorePasswd: truststorePassword
      nifi.cluster.protocol.is.secure: true
      nifi.cluster.is.node: true
      nifi.cluster.node.address: "{{ ansible_fqdn }}"
      nifi.cluster.node.protocol.port: 11443
      nifi.cluster.flow.election.max.candidates: 3
      nifi.cluster.load.balance.host: "{{ ansible_fqdn }}"
      nifi.cluster.load.balance.port: 6342
      nifi.state.management.embedded.zookeeper.start: true
      nifi.zookeeper.connect.string: nifi_server1:2181,nifi_server2:2181,nifi_server3:2181
    login_identity_providers:
      /loginIdentityProviders/provider/identifier: ldap-provider
      /loginIdentityProviders/provider/property[@name="Authentication Strategy"]: SIMPLE
      /loginIdentityProviders/provider/property[@name="Manager DN"]: cn=nifi,ou=people,dc=example,dc=com
      /loginIdentityProviders/provider/property[@name="Manager Password"]: password
      /loginIdentityProviders/provider/property[@name="Url"]: ldap://hostname:port
      /loginIdentityProviders/provider/property[@name="User Search Base"]: OU=people,DC=example,DC=com
      /loginIdentityProviders/provider/property[@name="User Search Filter"]: sAMAccountName={0}
    authorizers_identifiers:
      - file-user-group-provider
      - ldap-user-group-provider
      - composite-user-group-provider
    authorizers:
      /authorizers/userGroupProvider/property[@name="Initial User Identity 1"]: cn=John Smith,ou=people,dc=example,dc=com
      # Use the full DN of the node certificates here
      /authorizers/userGroupProvider/property[@name="Initial User Identity 2"]: CN=nifi_server1.example.com, O=ExampleLLC, L=Saint Louis, ST=Missouri, C=US
      /authorizers/userGroupProvider/property[@name="Initial User Identity 3"]: CN=nifi_server2.example.com, O=ExampleLLC, L=Saint Louis, ST=Missouri, C=US
      /authorizers/userGroupProvider/property[@name="Initial User Identity 4"]: CN=nifi_server3.example.com, O=ExampleLLC, L=Saint Louis, ST=Missouri, C=US
      /authorizers/accessPolicyProvider/property[@name="Initial Admin Identity"]: cn=John Smith,ou=people,dc=example,dc=com
      /authorizers/accessPolicyProvider/property[@name="Node Identity 1"]: CN=nifi_server1.example.com, O=ExampleLLC, L=Saint Louis, ST=Missouri, C=US
      /authorizers/accessPolicyProvider/property[@name="Node Identity 2"]: CN=nifi_server2.example.com, O=ExampleLLC, L=Saint Louis, ST=Missouri, C=US
      /authorizers/accessPolicyProvider/property[@name="Node Identity 3"]: CN=nifi_server3.example.com, O=ExampleLLC, L=Saint Louis, ST=Missouri, C=US
    state_management:
      /stateManagement/cluster-provider/property[@name="Connect String"]: "{{ nifi_properties['nifi.zookeeper.connect.string'] }}"
    # Assuming nifi_server1 = 192.168.1.10, nifi_server2 = 192.168.1.11, nifi_server3 = 192.168.1.12
    # we have Ansible automatically set the myid file on each host to last octet of the node's IP address
    # and we set the 'X' of the zookeeper['server.X'] keys to those same numbers.
    zookeeper_myid: "{{ ansible_default_ipv4.address.split('.')[-1] }}"
    zookeeper:
      server.10: nifi_server1:2888:3888
      server.11: nifi_server2:2888:3888
      server.12: nifi_server3:2888:3888

License

GPLv3

Author Information

This role was created in 2018 by cavemandaveman.