NAME
App::ElasticSearch::Utilities - Utilities for Monitoring ElasticSearch
VERSION
version 2.7
SYNOPSIS
This library contains utilities for unified interfaces in the scripts.
This a set of utilities to make monitoring ElasticSearch clusters much simpler.
Included are:
SEARCHING:
scripts/es-search.pl - Utility to interact with LogStash style indices from the CLI
MONITORING:
scripts/es-nagios-check.pl - Monitor ES remotely or via NRPE with this script
scripts/es-graphite-dynamic.pl - Perform index maintenance on daily indexes
scripts/es-status.pl - Command line utility for ES Metrics
scripts/es-storage-data.pl - View how shards/data is aligned on your cluster
scripts/es-nodes.pl - View node information
MAINTENANCE:
scripts/es-daily-index-maintenance.pl - Perform index maintenance on daily indexes
scripts/es-alias-manager.pl - Manage index aliases automatically
MANAGEMENT:
scripts/es-copy-index.pl - Copy an index from one cluster to another
scripts/es-apply-settings.pl - Apply settings to all indexes matching a pattern
scripts/es-storage-data.pl - View how shards/data is aligned on your cluster
DEPRECATED:
scripts/es-graphite-static.pl - Send ES Metrics to Graphite or Cacti
The App::ElasticSearch::Utilities module simply serves as a wrapper around the scripts for packaging and distribution.
FUNCTIONS
es_pattern
Returns a hashref of the pattern filter used to get the indexes { string => '*', re => '.*', }
es_connect
Without options, this connects to the server defined in the args. If passed an array ref, it will use that as the connection definition.
es_request([$handle],$command,{ method => 'GET', parameters => { a => 1 } }, {})
Retrieve URL from ElasticSearch, returns a hash reference
First hash ref contains options, including:
uri_param Query String Parameters
index Index name
type Index type
method Default is GET
es_nodes
Returns the hash of index meta data.
es_indices_meta
Returns the hash of index meta data.
es_indices
Returns a list of active indexes matching the filter criteria specified on the command line. Can handle indices named:
logstash-YYYY.MM.DD
dcid-logstash-YYYY.MM.DD
logstash-dcid-YYYY.MM.DD
logstash-YYYY.MM.DD-dcid
Makes use of --datesep to determine where the date is.
es_index_days_old( 'index-name' )
Return the number of days old this index is.
es_index_shard_replicas( 'index-name' )
Returns the number of replicas for a given index.
es_index_valid( 'index-name' )
Checks if the specified index is valid
es_close_index('index-name')
Closes an index
es_open_index('index-name')
Open an index
es_delete_index('index-name')
Deletes an index
es_optimize_index('index-name')
Optimize an index to a single segment per shard
es_index_segments( 'index-name' )
Exposes GET /$index/_segments
Returns the segment data from the index in hashref:
es_segment_stats($index)
Return the number of shards and segments in an index as a hashref
es_index_stats( 'index-name' )
Exposes GET /$index/_stats
Returns a hashref
es_settings()
Exposes GET /_settings
Returns a hashref
es_node_stats()
Exposes GET /_nodes/stats
Returns a hashref
es_facet_whitelist('field name')
Returns if the field is whitelisted
Facet whitelists must be set in a configuration file. Currently, the search path for config files is
/etc/es-utils.yaml
/etc/es-utils.yml
$ENV{HOME}/.es-utils.yaml
$ENV{HOME}/.es-utils.yml
This does mean that users can override the whitelist, but this is by design. If one of those files does not specify a facet_whitelist element as an array, the whitelist is not restricted.
Examples:
--- facet_whitelist:
- src_ip
- src_ip_country
- dst_ip
- dst_ip_country
- file
- filetype
- program
- status
- method
- protocol
def('key')
Exposes Definitions grabbed by options parsing
ARGS
From App::ElasticSearch::Utilities:
--local Use localhost as the elasticsearch host
--host ElasticSearch host to connect to
--port HTTP port for your cluster
--noop Any operations other than GET are disabled
--timeout Timeout to ElasticSearch, default 30
--keep-proxy Do not remove any proxy settings from %ENV
--index Index to run commands against
--base For daily indexes, reference only those starting with "logstash"
(same as --pattern logstash-* or logstash-DATE)
--datesep Date separator, default '.' also (--date-separator)
--pattern Use a pattern to operate on the indexes
--days If using a pattern or base, how many days back to go, default: all
ARGUMENT GLOBALS
Some options may be specified in the /etc/es-utils.yaml or $HOME/.es-utils.yaml file:
---
host: esproxy.example.com
port: 80
timeout: 10
INSTALL
This library attempts to provide scripts compatible with version 0.19 through 1.1 of ElasticSearch.
Recommended install with CPAN Minus:
cpanm App::ElasticSearch::Utilities
You can also use CPAN:
cpan App::ElasticSearch::Utilities
Or if you'd prefer to manually install:
export RELEASE=<CurrentRelease>
wget --no-check-certificate https://github.com/reyjrar/es-utils/blob/master/releases/App-ElasticSearch-Utilities-$RELEASE.tar.gz?raw=true -O es-utils.tgz
tar -zxvf es-utils.tgz
cd App-ElasticSearch-Utilities-$RELEASE
perl Makefile.PL
make
make install
This will take care of ensuring all the dependencies are satisfied and will install the scripts into the same directory as your Perl executable.
USAGE
The tools are all wrapped in their own documentation, please see:
$UTILITY --help
$UTILITY --manual
For individual options and capabilities
PATTERNS
Patterns are used to match an index to the aliases it should have. A few symbols are expanded into regular expressions. Those patterns are:
* expands to match any number of any characters.
? expands to match any single character.
DATE expands to match YYYY.MM.DD, YYYY-MM-DD, or YYYYMMDD
ANY expands to match any number of any characters.
CONTRIBUTORS
Mihai Oprea <mishu@mishulica.com>
Samit Badle
AUTHOR
Brad Lhotsky brad@divisionbyzero.net
COPYRIGHT AND LICENSE
This software is Copyright (c) 2012 by Brad Lhotsky.
This is free software, licensed under:
The (three-clause) BSD License