rabe-zabbix
Collection of various Zabbix templates and helper scripts created or used by Radio Bern RaBe.
Zabbix Templates
The Zabbix templates are located within the templates subfolder and are further subdived into the following categories:
- Operating system (os): All operating system specific templates
- Naming convention:
Template_OS_<OPERATING-SYSTEM-NAME>_active
- Example:
Template_OS_Linux_active
- Naming convention:
- Application (app): All application specifc templates
- Naming convention:
Template_App_[<OPERATING-SYSTEM-NAME]_<APPLICATION-NAME>_active
- Example:
Template_App_Linux_Postfix_active
- Naming convention:
- IPMI (ipmi): All IPMI based templates
- Naming convention:
Template_IPMI_<NAME>
- Example:
Template_IPMI_Threshold_Sensors
- Naming convention:
- SNMP (snmp): All SNMP based templates
- Naming convention:
Template_SNMP<SNMP-VERSION>_<NAME>
- Example:
Template_SNMPv2_Interfaces_HC
- Naming convention:
- Stack (stack): Stack templates define a single system (or role) and might inherit further stack templates, or exactly one operating system template as well as multiple application or SNMP templates.
- Naming convention:
Template_Stack_<STACK-NAME>_active
- Example:
Template_Stack_MediaWiki_active
Stack templates explained
Every host within Zabbix gets a specifc stack template assigned according to its role. The stack exactly defines the setup of this host and will be re-used if there is more than one host with the same role.
As an example, a host which servers a MediaWiki instance, will get the stack template Template_Stack_MediaWiki_active
assigned. The stack template might include the operating system template Template_OS_Linux_active
, the application templates Template_App_Linux_Apache_HTTPD_active
, Template_App_Linux_PHP-FPM_active
and Template_App_Linux_MariaDB_active
.
This ensures great modularity, reusability and avoids unecessary inheritance problems.
Further template conventions
- Use active mode for zabbix agent items by default
- Use an update interval of 300 seconds (5 minutes) by default
- Create at least one unique application per app, ipmi and snmp template
- Use macros whenever possible and feasible, prefix them with a unique per template prefix
App specific conventions
- Apps may contain configuration snippets in a
userparameters/
subdir. - SELinux policy modules for an app are in the
selinux/
subdir. They are prefixed with "rabezbx" to help differentiate them from system policy.
IPMI specific conventions
- Name server or motherboard templates according to
IPMI <VENDOR> <PRODUCT-NAME>
, for exampleTemplate IPMI Supermicro SSG-6048R-E1CR24N
- Try to build up a server or motherboard template from existing (or newly created) standalone sensor templates (which can be reused for the same sensor and reading type).
- Standalone sensor templates which contain sensor-specific discrete IPMI
sensors (event/reading type code 6Fh)
- Template naming according to
IPMI <SENSOR-NAME> Sensors
, for exampleIPMI Power Supply Sensors
- Item key naming according to
ipmi.discrete-sensor[<SENSOR-TYPE>,<SENSOR-NAME>]
, for exampleipmi.discrete-sensor[power-supply,{#IPMI_SENSOR_NAME}]
- Create triggers according to the sensor's specific event/reading type code and offsets (see Table 42-3, Sensor Type Codes) with the help of the Zabbix band() function
- Template naming according to
- Standalone sensor templates which contain generic discrete IPMI sensors
(event/reading type code 02h - 0Ch)
- Template naming according to
IPMI <SENSOR-NAME> Generic Sensors
, for exampleIPMI Module Board Generic Sensors
- Item key naming according to
ipmi.discrete-generic-sensor[<SENSOR-TYPE>,<SENSOR-NAME>]
, for exampleipmi.discrete-generic-sensor[module-board,{#IPMI_SENSOR_NAME}]
- Create triggers according to the sensor's generic event/reading type code and offsets (see Table 42-2, Generic Event/Reading Type Codes) with the help of the Zabbix band() function
- Template naming according to
- Use the provided
ipmi-sensor-discovery.sh
external check script for low-level auto-discovery of multiple sensors. - FreeIPMI's interpret sensor configuration might be helpful for mapping sensor states to Zabbix trigger severities.
- There should be no more need for threshold based sensor templates
(event/reading type code 01h), as they are already handled by the
IPMI Threshold Sensors
template.
SNMP specific conventions
- Name SNMP templates according to
Template SNMPv<SNMP-VERSION> <NAME>
, for exampleTemplate SNMPv2 Bridge
- Try to reflect the MIBs name within
<NAME>
whenever feasible - Use the textual form of MIB OIDs within your SNMP items.
As an example, use
BRIDGE-MIB::dot1dBaseNumPorts.0
instead of.1.3.6.1.2.1.17.1.2.0
- Include instructions on how to obtain and install the required MIBs for your template.
- Name items according to
rabe.snmp.<NAME>.<OID-NAME>
to avoid clashes with other templates. Example:rabe.snmp.bridge.dot1dBaseNumPorts
- Name low-level discovery rule keys with
rabe.snmp.<NAME>.<OBJECT>.discovery
Example:rabe.snmp-bridge.ports.discovery
- Create value mappings according to the OID's syntax definition from the MIB.
Developing
Adding an app template
appName="" # app name
appURL="" # URL to the upstream project/software
lowercaseName="${appName,,}"
uppercaseName="${appName^^}"
shortName="${lowercaseName//-/}"
templateName="Template App ${appName} active"
xmlName="${templateName// /_}.xml"
appDir="app/${appName// /_}"
gitBranchName="feature/app-${lowercaseName// /-}"
git checkout -b "${gitBranchName}"
mkdir -p "${appDir}/doc"
# Generate a minimal documentation
cat > "${appDir}/doc/README.head.md" << EOF
Monitoring of [${appName}]($appURL).
EOF
# Extend the documentation as necessary
vi "${appDir}/doc/README.head.md"
# Commit the documentation
git add "${appDir}/doc/README.head.md"
git commit -m "${appName}: Added documentation"
# Export the Zabbix template and move it to the final destination
# see fetching below if you want help exporting
mv zbx_export_templates.xml "${appDir}/${xmlName}"
# Commit the Zabbix template
git add "${appDir}/${xmlName}"
git commit -m "${appName}: Added ${templateName}"
Fetching an app from the Zabbix server
Fetching a template into a file from the server is straightforward.
./helper/rabe-zabbix --template=${templateName} ${xmlName}
You need to have a working node environment and install any dependencies beforehand.
npm install
The helper will prompt for settings should they not be configured. Please see its --help
for more information.
optional usage section override
cat > "${appDir}/doc/README.Usage.md" << EOF
## Usage
1. Import the
[\`${xmlName}\`](${xmlName})
into your Zabbix server (click on the \`Raw\` button to download).
2. Add the template to your host (or stack template)
3. Check if new data arrives
EOF
vi "${appDir}/doc/README.Usage.md"
git add "${appDir}/doc/README.Usage.md"
git commit -m "${appName}: Added usage section"
optional selinux policy
$avcViolation="" # multiple lines from /var/log/audit/audit.log
moduleName="rabezbx${shortName// /}"
mkdir "${appDir}/selinux"
echo ${avcViolation} | audit2allow -m ${moduleName} > \
"${appDir}/selinux/${moduleName}.te"
cat > "${appDir}/doc/README.SELinux.md" <<EOD
## SELinux Policy
The [${moduleName}](selinux/${moduleName}.te) policy does <dox>.
EOD
# Commit the SELinux policy and documentation
git add "${appDir}/selinux/${moduleName}.te"
git add "${appDir}/doc/README.SELinux.md"
git commit -e -m "${appName}: Added SELinux module and documentation"
optional userparameters
mkdir "${appDir}/userparameters"
userParameterName="rabe.${lowercaseName// /-}"
cat > "${appDir}/userparameters/${userParameterName}.conf" <<EOD
#
# dox here
#
UserParameter=${userParameterName}.<key>,<script>
EOD
cat > "${appDir}/doc/README.UserParameters.md" <<EOD
## UserParameters
| Key | Description |
| --- | ----------- |
| \`${userParameterName}.<key>\` | <dox> |
EOD
# Commit the userparameters and documentation
git add "${appDir}/userparameters/${userParameterName}.conf"
git add "${appDir}/doc/README.UserParameters.md"
git commit -e -m "${appName}: Added user parameters and docs."
optional scripts
mkdir "${appDir}/scripts"
scriptName="rabe-${lowercaseName// /-}.sh"
touch "${appDir}/scripts/${scriptName}"
cat > "${appDir}/doc/README.scripts.md" <<EOD
## Scripts
* [${scriptName}](./scripts/${scriptName}) for ${userParameterName}.<key> UserParameter
<dox below listing if needed>
EOD
# Commit the scripts and documentation
git add "${appDir}/scripts/*"
git add "${appDir}/doc/README.scripts.md"
git commit -e -m "${appName}: Added scripts and docs."
optional Sudo security policies
mkdir "${appDir}/sudoers.d"
sudoersFileName="rabezbx-${lowercaseName// /_}"
sudoersCmndAliasPrefix="RABEZBX_${uppercaseName//[ -]/_}"
cat > "${appDir}/sudoers.d/${sudoersFileName}" << EOF
##
## Defaults specification for the zabbix user
##
Defaults:zabbix !requiretty
##
## Command alias specifications for ${appName}
##
Cmnd_Alias ${sudoersCmndAliasPrefix}_MYCMD = /sbin/mycmd -a -b -c
Cmnd_Alias ${sudoersCmndAliasPrefix}_MYOTHERCMD = /sbin/myothercmd
##
## User privilege specifications for the zabbix user
##
zabbix ALL=NOPASSWD: ${sudoersCmndAliasPrefix}_MYCMD
zabbix ALL=NOPASSWD: ${sudoersCmndAliasPrefix}_MYOTHERCMD
EOF
# Commit the sudo configuration snippet
git add "${appDir}/sudoers.d/${sudoersFileName}"
git commit -e -m "${appName}: Added sudoers config."
Adapt the MYCMD
and MYOTHERMCD
command aliases accordingly.
Generate the template documentation and push
As a final step you have to generate the template configuration and push to GitHub afterwards.
# Generate the template documentation
make update-app-doc
git add "${appDir}/README.md"
git commit -m "${appName}: Added generated documentation"
# Push and create a PR on GitHub afterwards
git push --set-upstream origin "${gitBranchName}"
Adding an OS template
osName="" # Operating system name
lowercaseName="${osName,,}"
gitBranchName="feature/os-${lowercaseName// /-}"
git checkout -b "${gitBranchName}"
templateName="Template OS ${osName}"
xmlName="${templateName// /_}.xml"
osDir="os/${osName// /_}"
mkdir -p "${osDir}/doc"
touch "${osDir}/doc/README.head.md"
# Write the documentation for your template
vi "${osDir}/doc/README.head.md"
git add "${osDir}/doc/README.head.md"
git commit -m "${osName}: Added documentation"
# Export the Zabbix template and move it to the final destination
mv zbx_export_templates.xml "${osDir}/${xmlName}"
git add "${osDir}/${xmlName}"
git commit -m "${osName}: Added ${templateName}"
# Generate the template documentation
make update-os-doc
git add "${osDir}/README.md"
git commit -m "${osName}: Added generated documentation"
# Push and create a PR on GitHub afterwards
git push --set-upstream origin "${gitBranchName}"
optional usage section override
cat > "${osDir}/doc/README.Usage.md" << EOF
## Usage
EOF
vi "${osDir}/doc/README.Usage.md"
git add "${osDir}/doc/README.Usage.md"
git commit -m "${osName}: Added usage section"
Adding an IPMI template
ipmiName="" # IPMI sensor, board or server name
lowercaseName="${ipmiName,,}"
templateName="Template IPMI ${ipmiName}"
xmlName="${templateName// /_}.xml"
ipmiDir="ipmi/${ipmiName// /_}"
mkdir -p "${ipmiDir}/doc"
touch "${ipmiDir}/doc/README.head.md"
mv zbx_export_templates.xml "${ipmiDir}/${xmlName}"
optional usage section override
cat > "${ipmiDir}/doc/README.Usage.md" << EOF
## Usage
EOF
vi "${ipmiDir}/doc/README.Usage.md"
git add "${ipmiDir}/doc/README.Usage.md"
git commit -m "${ipmiName}: Added usage section"
optional scripts
scriptName="ipmi-${lowercaseName// /-}"
mkdir -p "${ipmiDir}/scripts"
touch "${ipmiDir}/scripts/${scriptName}.sh"
cat > "${ipmiDir}/doc/README.scripts.md" <<EOD
## Scripts
* [${scriptName}.sh](./scripts/${scriptName}.sh) <short description>
<dox below listing if needed>
EOD
Adding an SNMP template
snmpName="" # Name, usually related to the MIB or device
snmpVersion="2" # SNMP version, 1, 2 or 3.
snmpDeviceURL="" # URL to the device (if any)
snmpMibURL="" # URL to the MIB (if any)
lowercaseName="${snmpName,,}"
templateName="Template SNMPv${snmpVersion} ${snmpName}"
xmlName="${templateName// /_}.xml"
snmpDir="snmp/SNMPv${snmpVersion}_${snmpName// /_}"
gitBranchName="feature/snmp-${lowercaseName// /-}"
git checkout -b "${gitBranchName}"
mkdir -p "${snmpDir}/doc"
touch "${snmpDir}/doc/README.head.md"
# Generate a minimal documentation
cat > "${snmpDir}/doc/README.head.md" << EOF
<DEVICE-SPECIFIC-EXAMPLE>
Monitoring of [${snmpName}](${snmpDeviceURL})
devices via SNMPv${snmpVersion}
</DEVICE-SPECIFIC-EXAMPLE
<MIB-SPECIFIC-EXAMPLE>
Monitors <EXAMPLE> parameters exposed by the
[\`<EXAMPLE-MIB>\`](${snmpMibURL}) via SNMPv${snmpVersion}
</MIB-SPECIFIC-EXAMPLE>
EOF
cat > "${snmpDir}/doc/README.Usage.md" << EOF
## Usage
1. Download the [<EXAMPLE-MIB>](${snmpMibURL})
2. Place the MIB file(s) into your default MIB directory on the Zabbix server
and/or proxy (usually \`/usr/local/share/snmp/mibs\`) and make sure that the
Zabbix server and/or proxy loads them (see [Using and loading
MIBs](http://www.net-snmp.org/wiki/index.php/TUT:Using_and_loading_MIBS)).
3. Import the
[\`${xmlName}\`](${xmlName})
into your Zabbix server (click on the \`Raw\` button to download).
4. Add an SNMP interface configuration to your host
5. Set the `{$SNMP_COMMUNITY}` macro to your desired community if you don't use
`public`
6. Add the template to your host (or stack template)
7. Check if new data arrives
## Notes
### snmpwalk command
The following \`snmpwalk\` command might be helpful for debugging:
\`\`\`bash
# Include and adapt me for SNMP version 1 or 2c
snmpwalk -v <1|2c> -c public <HOST> <EXAMPLE-MIB::OID>
# Include and adapt me for SNMP version 3
snmpwalk -v 3 -l <noAuthNo‐Priv|authNoPriv|authPriv> \\
[-a <MD5|SHA> -A <AUTH-PASSPHRASE> \\
[-x <DES|AES> -X <PRIV-PASSPHRASE>]] \\
<HOST> <EXAMPLE-MIB::OID>
\`\`\`
EOF
# Adapt and extend the documentation as necessary
vi "${snmpDir}/doc/README.head.md"
vi "${snmpDir}/doc/README.Usage.md"
# Commit the documentation
git add "${snmpDir}/doc/README.head.md"
git add "${snmpDir}/doc/README.Usage.md"
git commit -m "${snmpName}: Added documentation"
# Export the Zabbix template and move it to its final destination
mv zbx_export_templates.xml "${snmpDir}/${xmlName}"
# Commit the Zabbix template
git add "${snmpDir}/${xmlName}"
git commit -m "${snmpName}: Added ${templateName}"
# Generate the template documentation
make update-snmp-doc
git add "${snmpDir}/README.md"
git commit -m "${snmpName}: Added generated documentation"
# Push and create a PR on GitHub afterwards
git push --set-upstream origin "${gitBranchName}"
Note, that you can also use the provided template fetching helper script for downloading the template from your Zabbix server.
Debugging
The following commands might be helpful for general debugging:
- Test a specific Zabbix agent item
su -c 'zabbix_agentd -t <ITEM-KEY>' -s /bin/bash zabbix
- Restart the Zabbix agent
systemctl restart zabbix-agent
- Restart the Zabbix server
systemctl restart zabbix-server
The following logs might contain helpful hints:
- Zabbix Agent related messages
/var/log/zabbix/zabbix_agentd.log
- Zabbix Server related messages
/var/log/zabbix/zabbix_server.log
- sudo related messages
journalctl -r /usr/bin/sudo
- SELinux related messages
/var/log/audit/audit.log
RPM Packages
The rabe-zabbix templates come with an RPM package that helps install SELinux policies and UserParameter configs. We provide a pre-built version through the openSUSE Build Server. They are available as part of the home:radiorabe:zabbix Subproject. You can install them as follows.
curl -o /etc/yum.repos.d/home:radiorabe:zabbix.repo \
http://download.opensuse.org/repositories/home:/radiorabe:/zabbix/CentOS_7/home:radiorabe:zabbix.repo
yum install rabe-zabbix
Releasing
- Bump the version in the Specfile and add a new commit to master with the version bump
- Tag this commit with the same version you used in the Specfile
- Push master and the tag to github
- The openSUSE Build Service should get triggered and build a new package automagically
License
This template collection is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, version 3 of the License.
Copyright
Copyright (c) 2017 - 2019 Radio Bern RaBe