- Module description - What the module does and why it is useful
- Setup - The basics of getting started with Opensearch
- Usage - Configuration options and additional functionality
- Advanced features - Extra information on advanced usage
- Reference - An under-the-hood peek at what the module is doing and how
- Limitations - OS compatibility, etc.
- Development - Guide for contributing to the module
- Support - When you need help with this module
- Transfer Notice
While waiting officail module from OpenSearch community: opensearch-project/opensearch-devops#67 Fork from official puppet mudule for ElasticSearch made by just seding, renaming elasticsearch -> opensearch. Basically works (most of basic functions), partialy not (for example template management).
This module sets up Opensearch instances with additional resource for plugins, templates, and more.
This module is actively tested against Opensearch 2.0.0
Therefore please ensure that you test this major release in your environment before using it in production!
- Opensearch repository files.
- Opensearch package.
- Opensearch configuration file.
- Opensearch service.
- Opensearch plugins.
- Opensearch snapshot repositories.
- Opensearch templates.
- Opensearch ingest pipelines.
- Opensearch index settings.
- Opensearch users, roles, and certificates.
- Opensearch licenses.
- Opensearch keystores.
- The stdlib Puppet library.
- richardc/datacat
- Augeas
- puppetlabs-java_ks for certificate management (optional).
Beginning with Opensearch 7.0.0, a Java JDK has been bundled as part of the opensearch package. However there still needs to be a version of Java present on the system being managed in order for Puppet to be able to run various utilities. We recommend managing your Java installation with the puppetlabs-java module.
When using the repository management, the following module dependencies are required:
- General: Puppet/elastic_stack
- Debian/Ubuntu: Puppetlabs/apt
- openSUSE/SLES: puppet/zypprepo
Declare the top-level opensearch class (managing repositories) and set up an instance:
include ::java
class { 'opensearch': }Most top-level parameters in the opensearch class are set to reasonable defaults.
The following are some parameters that may be useful to override:
class { 'opensearch':
version => '7.9.3'
}Note: This will only work when using the repository.
By default, the module will not restart Opensearch when the configuration file, package, or plugins change. This can be overridden globally with the following option:
class { 'opensearch':
restart_on_change => true
}Or controlled with the more granular options: restart_config_change, restart_package_change, and restart_plugin_change.
class { 'opensearch':
autoupgrade => true
}class { 'opensearch':
ensure => 'absent'
}class { 'opensearch':
status => 'disabled'
}Some resources, such as opensearch::template, require communicating with the Opensearch REST API.
By default, these API settings are set to:
class { 'opensearch':
api_protocol => 'http',
api_host => 'localhost',
api_port => 9200,
api_timeout => 10,
api_basic_auth_username => undef,
api_basic_auth_password => undef,
api_ca_file => undef,
api_ca_path => undef,
validate_tls => true,
}Each of these can be set at the top-level opensearch class and inherited for each resource or overridden on a per-resource basis.
This module supports managing all of its defined types through top-level parameters to better support Hiera and Puppet Enterprise.
For example, to manage an index template directly from the opensearch class:
class { 'opensearch':
templates => {
'logstash' => {
'content' => {
'template' => 'logstash-*',
'settings' => {
'number_of_replicas' => 0
}
}
}
}
}This module can help manage a variety of plugins.
Note that module_dir is where the plugin will install itself to and must match that published by the plugin author; it is not where you would like to install it yourself.
opensearch::plugin { 'x-pack': }opensearch::plugin { 'jetty':
url => 'https://oss-es-plugins.s3.amazonaws.com/opensearch-jetty/opensearch-jetty-1.2.1.zip'
}You can also use a proxy if required by setting the proxy_host and proxy_port options:
opensearch::plugin { 'lmenezes/opensearch-kopf',
proxy_host => 'proxy.host.com',
proxy_port => 3128
}Proxies that require usernames and passwords are similarly supported with the proxy_username and proxy_password parameters.
Plugin name formats that are supported include:
opensearch/plugin/version(for official opensearch plugins downloaded from download.elastic.co)groupId/artifactId/version(for community plugins downloaded from maven central or OSS Sonatype)username/repository(for site plugins downloaded from github master)
When you specify a certain plugin version, you can upgrade that plugin by specifying the new version.
opensearch::plugin { 'opensearch/opensearch-cloud-aws/2.1.1': }And to upgrade, you would simply change it to
opensearch::plugin { 'opensearch/opensearch-cloud-aws/2.4.1': }Please note that this does not work when you specify 'latest' as a version number.
For the Opensearch commercial plugins you can refer them to the simple name.
See Plugin installation for more details.
Installs scripts to be used by Opensearch. These scripts are shared across all defined instances on the same host.
opensearch::script { 'myscript':
ensure => 'present',
source => 'puppet:///path/to/my/script.groovy'
}Script directories can also be recursively managed for large collections of scripts:
opensearch::script { 'myscripts_dir':
ensure => 'directory,
source => 'puppet:///path/to/myscripts_dir'
recurse => 'remote',
}By default templates use the top-level opensearch::api_* settings to communicate with Opensearch.
The following is an example of how to override these settings:
opensearch::template { 'templatename':
api_protocol => 'https',
api_host => $::ipaddress,
api_port => 9201,
api_timeout => 60,
api_basic_auth_username => 'admin',
api_basic_auth_password => 'adminpassword',
api_ca_file => '/etc/ssl/certs',
api_ca_path => '/etc/pki/certs',
validate_tls => false,
source => 'puppet:///path/to/template.json',
}This will install and/or replace the template in Opensearch:
opensearch::template { 'templatename':
source => 'puppet:///path/to/template.json',
}This will install and/or replace the template in Opensearch:
opensearch::template { 'templatename':
content => {
'template' => "*",
'settings' => {
'number_of_replicas' => 0
}
}
}Plain JSON strings are also supported.
opensearch::template { 'templatename':
content => '{"template":"*","settings":{"number_of_replicas":0}}'
}opensearch::template { 'templatename':
ensure => 'absent'
}Pipelines behave similar to templates in that their contents can be controlled
over the Opensearch REST API with a custom Puppet resource.
API parameters follow the same rules as templates (those settings can either be
controlled at the top-level in the opensearch class or set per-resource).
This will install and/or replace an ingestion pipeline in Opensearch (ingestion settings are compared against the present configuration):
opensearch::pipeline { 'addfoo':
content => {
'description' => 'Add the foo field',
'processors' => [{
'set' => {
'field' => 'foo',
'value' => 'bar'
}
}]
}
}opensearch::pipeline { 'addfoo':
ensure => 'absent'
}This module includes basic support for ensuring an index is present or absent with optional index settings. API access settings follow the pattern previously mentioned for templates.
At the time of this writing, only index settings are supported.
Note that some settings (such as number_of_shards) can only be set at index
creation time.
opensearch::index { 'foo':
settings => {
'index' => {
'number_of_replicas' => 0
}
}
}opensearch::index { 'foo':
ensure => 'absent'
}By default snapshot_repositories use the top-level opensearch::api_* settings to communicate with Opensearch.
The following is an example of how to override these settings:
opensearch::snapshot_repository { 'backups':
api_protocol => 'https',
api_host => $::ipaddress,
api_port => 9201,
api_timeout => 60,
api_basic_auth_username => 'admin',
api_basic_auth_password => 'adminpassword',
api_ca_file => '/etc/ssl/certs',
api_ca_path => '/etc/pki/certs',
validate_tls => false,
location => '/backups',
}opensearch::snapshot_repository { 'backups':
ensure => 'absent',
location => '/backup'
}This module offers a way to make sure an instance has been started and is up and running before
doing a next action. This is done via the use of the es_instance_conn_validator resource.
es_instance_conn_validator { 'myinstance' :
server => 'es.example.com',
port => '9200',
}A common use would be for example :
class { 'kibana4' :
require => Es_Instance_Conn_Validator['myinstance'],
}There are two different ways of installing Opensearch:
This module uses the elastic/elastic_stack module to manage package repositories. Because there is a separate repository for each major version of the Elastic stack, selecting which version to configure is necessary to change the default repository value, like this:
class { 'elastic_stack::repo':
version => 6,
}
class { 'opensearch':
version => '6.8.12',
}This module defaults to the upstream package repositories, which as of Opensearch 6.3, includes X-Pack. In order to use the purely OSS (open source) package and repository, the appropriate oss flag must be set on the elastic_stack::repo and opensearch classes:
class { 'elastic_stack::repo':
oss => true,
}
class { 'opensearch':
oss => true,
}You may want to manage repositories manually. You can disable automatic repository management like this:
class { 'opensearch':
manage_repo => false,
}When a repository is not available or preferred you can install the packages from a remote source:
class { 'opensearch':
package_url => 'https://download.opensearch.org/opensearch/opensearch/opensearch-1.4.2.deb',
proxy_url => 'http://proxy.example.com:8080/',
}Setting proxy_url to a location will enable download using the provided proxy
server.
This parameter is also used by opensearch::plugin.
Setting the port in the proxy_url is mandatory.
proxy_url defaults to undef (proxy disabled).
class { 'opensearch':
package_url => 'puppet:///path/to/opensearch-1.4.2.deb'
}class { 'opensearch':
package_url => 'file:/path/to/opensearch-1.4.2.deb'
}When configuring Opensearch's memory usage, you can modify it by setting jvm_options:
class { 'opensearch':
jvm_options => [
'-Xms4g',
'-Xmx4g'
]
}Currently only the basic SysV-style init and Systemd service providers are supported, but other systems could be implemented as necessary (pull requests welcome).
The defaults file (/etc/defaults/opensearch or /etc/sysconfig/opensearch) for the Opensearch service can be populated as necessary.
This can either be a static file resource or a simple key value-style hash object, the latter being particularly well-suited to pulling out of a data source such as Hiera.
class { 'opensearch':
init_defaults_file => 'puppet:///path/to/defaults'
}$config_hash = {
'ES_HEAP_SIZE' => '30g',
}
class { 'opensearch':
init_defaults => $config_hash
}Note: init_defaults hash can be passed to the main class and to the instance.
File-based users, roles, and certificates can be managed by this module.
Note: If you are planning to use these features, it is highly recommended you read the following documentation to understand the caveats and extent of the resources available to you.
Roles in the file realm can be managed using the opensearch::role type.
For example, to create a role called myrole, you could use the following resource:
opensearch::role { 'myrole':
privileges => {
'cluster' => [ 'monitor' ],
'indices' => [{
'names' => [ '*' ],
'privileges' => [ 'read' ],
}]
}
}This role would grant users access to cluster monitoring and read access to all indices.
See the Security documentation for your version to determine what privileges to use and how to format them (the Puppet hash representation will simply be translated into yaml.)
Note: The Puppet provider for opensearch_user has fine-grained control over the roles.yml file and thus will leave the default roles in-place.
If you would like to explicitly purge the default roles (leaving only roles managed by puppet), you can do so by including the following in your manifest:
resources { 'opensearch_role':
purge => true,
}Associating mappings with a role for file-based management is done by passing an array of strings to the mappings parameter of the opensearch::role type.
For example, to define a role with mappings:
opensearch::role { 'logstash':
mappings => [
'cn=group,ou=devteam',
],
privileges => {
'cluster' => 'manage_index_templates',
'indices' => [{
'names' => ['logstash-*'],
'privileges' => [
'write',
'delete',
'create_index',
],
}],
},
}If you'd like to keep the mappings file purged of entries not under Puppet's control, you should use the following resources declaration because mappings are a separate low-level type:
resources { 'opensearch_role_mapping':
purge => true,
}Users can be managed using the opensearch::user type.
For example, to create a user mysuser with membership in myrole:
opensearch::user { 'myuser':
password => 'mypassword',
roles => ['myrole'],
}The password parameter will also accept password hashes generated from the esusers/users utility and ensure the password is kept in-sync with the Shield users file for all Opensearch instances.
opensearch::user { 'myuser':
password => '$2a$10$IZMnq6DF4DtQ9c4sVovgDubCbdeH62XncmcyD1sZ4WClzFuAdqspy',
roles => ['myrole'],
}Note: When using the esusers/users provider (the default for plaintext passwords), Puppet has no way to determine whether the given password is in-sync with the password hashed by Opensearch.
In order to work around this, the opensearch::user resource has been designed to accept refresh events in order to update password values.
This is not ideal, but allows you to instruct the resource to change the password when needed.
For example, to update the aforementioned user's password, you could include the following your manifest:
notify { 'update password': } ~>
opensearch::user { 'myuser':
password => 'mynewpassword',
roles => ['myrole'],
}SSL/TLS can be enabled by providing the appropriate class params with paths to the certificate and private key files, and a password for the keystore.
class { 'opensearch' :
ssl => true,
ca_certificate => '/path/to/ca.pem',
certificate => '/path/to/cert.pem',
private_key => '/path/to/key.pem',
keystore_password => 'keystorepassword',
}Note: Setting up a proper CA and certificate infrastructure is outside the scope of this documentation, see the aforementioned security guide for more information regarding the generation of these certificate files.
The module will set up a keystore file for the node to use and set the relevant options in opensearch.yml to enable TLS/SSL using the certificates and key provided.
System keys can be passed to the module, where they will be placed into individual instance configuration directories.
This can be set at the opensearch class and inherited across all instances:
class { 'opensearch':
system_key => 'puppet:///path/to/key',
}If you use the aforementioned security features, you may need to install a user license to leverage particular features outside of a trial license.
This module can handle installation of licenses without the need to write custom exec or curl code to install license data.
You may instruct the module to install a license through the opensearch::license parameter:
class { 'opensearch':
license => $license,
}The license parameter will accept either a Puppet hash representation of the license file json or a plain json string that will be parsed into a native Puppet hash.
Although dependencies are automatically created to ensure that the Opensearch service is listening and ready before API calls are made, you may need to set the appropriate api_* parameters to ensure that the module can interact with the Opensearch API over the appropriate port, protocol, and with sufficient user rights to install the license.
The native provider for licenses will not print license signatures as part of Puppet's changelog to ensure that sensitive values are not included in console output or Puppet reports.
Any fields present in the license parameter that differ from the license installed in a cluster will trigger a flush of the resource and new POST to the Opensearch API with the license content, though the sensitive signature field is not compared as it is not returned from the Opensearch licensing APIs.
There are several different ways of setting data directories for Opensearch.
In every case the required configuration options are placed in the opensearch.yml file.
By default we use:
/var/lib/opensearch
Which mirrors the upstream defaults.
It is possible to override the default data directory by specifying the datadir param:
class { 'opensearch':
datadir => '/var/lib/opensearch-data'
}It's also possible to specify multiple data directories using the datadir param:
class { 'opensearch':
datadir => [ '/var/lib/es-data1', '/var/lib/es-data2']
}See the Opensearch documentation for additional information regarding this configuration.
The config option can be used to provide additional configuration options to Opensearch.
The config hash can be written in 2 different ways:
Instead of writing the full hash representation:
class { 'opensearch':
config => {
'cluster' => {
'name' => 'ClusterName',
'routing' => {
'allocation' => {
'awareness' => {
'attributes' => 'rack'
}
}
}
}
}
}class { 'opensearch':
config => {
'cluster' => {
'name' => 'ClusterName',
'routing.allocation.awareness.attributes' => 'rack'
}
}
}Recent versions of Opensearch include the opensearch-keystore utility to create and manage the opensearch.keystore file which can store sensitive values for certain settings.
The settings and values for this file can be controlled by this module.
Settings follow the behavior of the config parameter for the top-level Opensearch class and opensearch::instance defined types.
That is, you may define keystore settings globally, and all values will be merged with instance-specific settings for final inclusion in the opensearch.keystore file.
Note that each hash key is passed to the opensearch-keystore utility in a straightforward manner, so you should specify the hash passed to secrets in flattened form (that is, without full nested hash representation).
For example, to define cloud plugin credentials for all instances:
class { 'opensearch':
secrets => {
'cloud.aws.access_key' => 'AKIA....',
'cloud.aws.secret_key' => 'AKIA....',
}
}By default, if a secret setting exists on-disk that is not present in the secrets hash, this module will leave it intact.
If you prefer to keep only secrets in the keystore that are specified in the secrets hash, use the purge_secrets boolean parameter either on the opensearch class to set it globally or per-instance.
Any changes to keystore secrets will notify running opensearch services by respecting the restart_on_change and restart_config_change parameters.
Class parameters are available in the auto-generated documentation pages. Autogenerated documentation for types, providers, and ruby helpers is also available on the same documentation site.
This module is built upon and tested against the versions of Puppet listed in the metadata.json file (i.e. the listed compatible versions on the Puppet Forge).
The module has been tested on:
- Amazon Linux 1/2
- Debian 8/9/10
- CentOS 7/8
- OracleLinux 7/8
- Ubuntu 16.04, 18.04, 20.04
- SLES 12
Testing on other platforms has been light and cannot be guaranteed.
Please see the CONTRIBUTING.md file for instructions regarding development environments and testing.
The Puppet Opensearch module is community supported and not officially supported by Elastic Support.
This module was originally authored by Elastic. The maintainer preferred that Vox Pupuli take ownership of the module for future improvement and maintenance. Existing pull requests and issues were transferred over, please fork and continue to contribute here instead of Elastic.