CESNET / Nemea-Supervisor

Management and monitoring module for the Nemea system.

README outline

• Project status
• Brief description
• How to build
• Dependencies
• Program parameters
• Program modes
• Configuration
  • Configuration file
  • Reload configuration
  • Installed configuration
• Supervisor functions
• Monitoring Nemea modules
• Log files
• Program termination
  • Backup file
  • Signals
• Supervisor client
  • Clients parameters
  • Collecting statistics about modules
  • Collecting information about modules

Project status

Travis CI build:

Coverity Scan:

Supervisor

This module allows user to configure and monitor Nemea modules (see basic modules and detectors). User specifies modules in xml configuration file, which is input for the Supervisor and it is shown and described in the section "Configuration file".

How to build

Simply with:

./bootstrap.sh
./configure
make

Useful parameters for configure are --prefix, --libdir, --sysconfdir and --bindir. For example:

./configure --sysconfdir=/etc/nemea/ --prefix=/usr/ --bindir=/usr/bin/nemea/ --libdir=/usr/lib64/

And possible installation:

sudo make install

Dependencies

Supervisor needs the following to be installed:

• libxml2 devel package

• Nemea-framework.

Program parameters

List of mandatory parameters the program accepts:

• -T FILE or --config-template=path Path of the XML file with the configuration template.

• -L PATH or --logs-path=path Path of the directory where the logs (both supervisor's and modules') will be saved.

List of optional parameters the program accepts:

• -d or --daemon Runs supervisor as a system daemon.

• -h or --help Prints program help.

• -s SOCKET or --daemon-socket=path Path of the unix socket which is used for supervisor daemon and client communication.

• -C PATH or --configs-path=path Path of the directory where the generated configuration files will be saved.

Program modes

Supervisor can run in one of the following modes:

1) Interactive Mode

supervisor -T supervisor_config_template.xml -L logs_path

2) System daemon

Program is executed with -d (or --daemon) argument and runs as a process in backround.

supervisor -T supervisor_config_template.xml -L logs_path --daemon

The activity of the running daemon can be controlled by a special thin client:

supervisor_cli (after installation from RPM, there is symlink "supcli" for the client)

Both daemon and client supports optional -s to specify path to the UNIX socket that is used for communication between daemon and client (more information about the client can be found in the section "Supervisor client").

3 ) System service

Supervisor is installed as a systemd service with the following commands:

• service nemea-supervisor start - starts the supervisor as a system deamon

• service nemea-supervisor stop - stops the deamon and all running Nemea modules from its configuration

• service nemea-supervisor restart - performs service start and service stop

• service nemea-supervisor status - returns running or stopped

• service nemea-supervisor reload - updates the configuration according to the configuration file

Configuration

Example

Example configuration file with comments: config_example.xml

The picture below should help to understand the configuration file. It consists of several module groups (profiles) which can be controlled separately.

Real usage of the configuration file

It is split into smaller parts called sup files (file.sup) for easier maintaining by multiple users. Example of such a file is this sup file containing a configuration of one Nemea detection module called dns tunnel detector.

Sup files can be placed into directories according to their category (e.g. detectors, data-sources etc.). It is shown here.

XML template helps to gather all the sup files and to construct the final configuration file using following directives:

<!-- include path_to_sup_file -->
<!-- include path_to_sup_files_dir -->

Path to the XML template is one of mandatory parameters for the supervisor (-T FILE or --config-template=path).

Reload configuration

The most important functionality of the supervisor. It updates the configuration according to the configuration file. It has the following phases:

• Generate config - generates the final configuration file by replacing include directives in XML template with the content of the .sup files

• Validate config - checks the generated config file according to defined syntax and semantic rules (structure and values)

• Apply config - if the validation successfully finishes, all changes are applied to the running configuration

There are three basic cases that can occur during "Apply config" phase:

• A module with same name was not found in loaded configuration -> a new module is inserted.

• A module with same name was found in loaded configuration -> every value is compared and if there is a difference, the module is reloaded.

• A module in loaded configuration was not found in the new configuration -> it is removed.

Installed configuration

After installation from RPM or by using

./configure --sysconfdir=/etc/nemea/ --prefix=/usr/ --bindir=/usr/bin/nemea/ --libdir=/usr/lib64/

during build (see How to build), there are 2 important paths with installed configuration.

• /usr/share/nemea-supervisor - contains default installed configuration from configs (all .sup files divided into directories the same way as on the picture here)

• /etc/nemea - contains XML template with includes of directories also from /etc/nemea.

It is possible to copy directories with .sup files from /usr/share/nemea-supervisor to /etc/nemea and use default configuration or it is possible to prepare own configuration and add the path of the directory or .sup file to the XML template.

Supervisor functions

User can do various operations with modules via Supervisor. After launch (either supervisor in non-daemon mode or supervisor_cli) a menu with the following operations appears:

• 1. ENABLE MODULE OR PROFILE - prints a list of disabled modules and profiles, the selected ones are enabled

• 2. DISABLE MODULE OR PROFILE - prints a list of enabled modules and profiles, the selected ones are disabled

• 3. RESTART RUNNING MODULE - prints a list of running modules, the selected ones are restarted

• 4. PRINT BRIEF STATUS - prints a table of the loaded modules divided into profiles and shows enabled, status and PID of all modules

• 5. PRINT DETAILED STATUS - prints the same information as the option 4 plus status of all modules interfaces including service interface (whether it is connected or number of connected clients)

• 6. PRINT LOADED CONFIGURATION - prints all information from the configuration file

• 7. RELOAD CONFIGURATION - performs reload of the configuration (see Reload configuration)

• 8. PRINT SUPERVISOR INFO - basic information about running supervisor - package and git version, used paths etc.

• 9. BROWSE LOG FILES - prints a list of all log files, selected log file is opened with pager (see Log files)

• 0. STOP SUPERVISOR - stops the supervisor but not the running modules

If the supervisor is running as a system daemon, last option "STOP SUPERVISOR" is replaced with

• -- Type "Cquit" to exit client -- - stops and disconnects the client

• -- Type "Dstop" to stop daemon -- - stops the supervisor but not the running modules

Monitoring Nemea modules

Modules status

Supervisor monitors the status of every module. The status can be running or stopped and it depends on the enabled flag of the module. Once the module is set to enabled, supervisor will automatically start it. If the module stops but is still enabled (user did not disable it), supervisor will restart it. Maximum number of restarts per minute can be specified with module-restarts in the configuration file. When the limit is reached, module is automatically set to disabled. If the module is running and it is disabled by user, SIGINT is used to stop the module. If it keeps running, SIGKILL must be used.

Statistics about modules´ interfaces

Every Nemea module has an implicit service interface, which allows Supervisor to get statistics about modules interfaces. These statistics include the following counters:

• for every input interface: received messages, recevied buffers

• for every output interface: sent messages, sent buffers, dropped messages, autoflushes

Data sent via service interface are encoded in JSON format. More information about service interface here

CPU and memory usage

The last monitored statistic is CPU usage (kernel and user mode) and system memory usage of every module.

Supervisor modules configuration

There is also a monitoring plugin to check that the supervisor could parse its modules configuration, see check_nemea_loaded_module_config.

Modules connection status

Among the information the supervisor extracts from each module is wether its input interfaces are connected, use the monitoring plugin check_nemea_modules_connected to keep track of this status.

Log files

Supervisor uses log files for its own output and also for an output of the started modules. Path of the logs can be specified with mandatory program parameter -L PATH or --logs-path=path.

Logs directory has the following content:

• supervisor_log - contains warning or error messages of the supervisor

• modules_events - contains messages about modules´ status changes

• modules_statistics - contains statistics about modules´ interfaces (they are printed periodically every minute)

• directory modules_logs - contains files with modules´ stdout and stderr in form of [module_name]_stdout and [module_name]_stderr

Program termination

Supervisor can be terminated via one of the menu options:

• interactive mode - option 0. STOP SUPERVISOR

• daemon mode - option Dstop

These two options don´t stop running modules and generate backup file (see next subsection).

To stop both the supervisor and all running modules, use service nemea-supervisor stop.

It is also possible to terminate the supervisor via sending a signal. service nemea-supervisor stop sends a SIGTERM. See all signals and their effect here

Backup file

Supervisor is able to terminate without stopping running modules and it will "find" them again after restart. This is achived via a backup file which is generated during termination if needed. Every backup file is bound to unique configuration template it was started with (-T parameter) and the path of the backup file is chosen according to it. The path is /tmp/sup_tmp_dir/PREFIX_sup_backup_file.xml where "PREFIX" is a number computed from absolute path of the configuration template. The backup file contains current configuration with PIDs of running modules. Together with backup file is also generated info file with basic information about current configuration.

For example supervisor started with -T /etc/nemea/supervisor_config_template.xml generates the following files:

• /tmp/sup_tmp_dir/89264_sup_backup_file.xml - backup file

• /tmp/sup_tmp_dir/89264_sup_backup_file.xml_info - file with basic information about current configuration

If the user executes supervisor with the same configuration template, supervisor will automatically find the backup file, loads the configuration and connects to running modules.

Signals

Signal handler catches the following signals:

• SIGTERM - stops all running modules, does not generate backup file (the only case modules are stopped)

• SIGINT - it let the modules continue running and generates backup file

• SIGQUIT - same as SIGINT

• SIGSEGV - this is for case something goes wrong during runtime. SIGSEGV is catched, modules continue running and backup file is saved.

Supervisor client

Clients parameters

List of optional parameters the program accepts:

• -h Prints program help.

• -s SOCKET Path of the unix socket which is used for supervisor daemon and client communication.

• -x Receives and prints statistics about modules and terminates.

• -r Sends a command to supervisor to reload the configuration.

• -i Receives and prints information about modules in JSON and terminates.

Note: All these parameters are optional so if the client is started without -x, -r or -i (supervisor_cli or supcli from RPM installation) it enters configuration mode with these functions.

Collecting statistics about modules

Supervisor client has a special mode that is enabled by -x. It allows user to get statistics about modules mentioned here in JSON format. In -x mode, client connects to the supervisor, receives and prints statistics, disconnects and terminates.

Note: this mode is used by plugin nemea-supervisor for munin.

Output notes

• interface type is one of {t, u, f, g, b} values corresponding to {tcpip, unix-socket, file, generator, blackhole}

• interface ID is port number (tcpip), socket name (unix-socket), file name (file) or "none" (generator, blackhole)

• interface counters are described here

Overall example of the output with statistics (reformatted):

{"haddrscan_detector": {"CPU-s": 0,
                        "CPU-u": 0,
                        "MEM-rss": 15396864,
                        "MEM-vms": 226459648,
                        "inputs": [{"ID": "egress_flow_data_source",
                                    "buffers": 412139,
                                    "is-conn": 1,
                                    "messages": 412139,
                                    "type": "u"}],
                        "outputs": [{"ID": "haddrscan_alerts",
                                     "autoflush": 4398,
                                     "buffers": 0,
                                     "cli-num": 1,
                                     "drop-msg": 0,
                                     "sent-msg": 0,
                                     "type": "u"}]},
 "vportscan_detector": {"CPU-s": 0,
                        "CPU-u": 0,
                        "MEM-rss": 15392768,
                        "MEM-vms": 226455552,
                        "inputs": [{"ID": "egress_flow_data_source",
                                    "buffers": 412154,
                                    "is-conn": 1,
                                    "messages": 412154,
                                    "type": "u"}],
                        "outputs": [{"ID": "vportscan_alerts",
                                     "autoflush": 4398,
                                     "buffers": 0,
                                     "cli-num": 1,
                                     "drop-msg": 0,
                                     "sent-msg": 0,
                                     "type": "u"}]}}

Collecting information about modules

Another special mode of the supervisor client is enabled by -i. It allows user to get basic information about modules in JSON format:

• module name (name of the running process)

• module index (in supervisor´s configuration)

• module parameters

• binary path

• module status (running or stopped)

• information about input and output interfaces

  • input ifc info format: ifc_type:ifc_id:is_conn where is connected is one char 0 or 1

  • output ifc info format: ifc_type:ifc_id:num_clients where number of clients is int32

In -i mode, client connects to the supervisor, receives and prints information, disconnects and terminates.

To see this data more clearly laid out and select only data for one or more modules the nemea-modulesinfo command is available.

Example of the output with modules information (reformatted):

{"haddrscan_detector": {"CPU-s": 0,
                        "CPU-u": 0,
                        "MEM-rss": 15396864,
                        "MEM-vms": 226459648,
                        "idx": 5,
                        "inputs": [{"ID": "egress_flow_data_source",
                                    "buffers": 382585,
                                    "is-conn": 1,
                                    "messages": 382585,
                                    "type": "u"}],
                        "outputs": [{"ID": "haddrscan_alerts",
                                     "autoflush": 4082,
                                     "buffers": 0,
                                     "cli-num": 1,
                                     "drop-msg": 0,
                                     "sent-msg": 0,
                                     "type": "u"}],
                        "params": "-i u:egress_flow_data_source,u:haddrscan_alerts",
                        "path": "/usr/bin/nemea/haddrscan_detector",
                        "status": "running"},
 "vportscan_detector": {"CPU-s": 0,
                        "CPU-u": 0,
                        "MEM-rss": 15392768,
                        "MEM-vms": 226455552,
                        "idx": 4,
                        "inputs": [{"ID": "egress_flow_data_source",
                                    "buffers": 382600,
                                    "is-conn": 1,
                                    "messages": 382600,
                                    "type": "u"}],
                        "outputs": [{"ID": "vportscan_alerts",
                                     "autoflush": 4082,
                                     "buffers": 0,
                                     "cli-num": 1,
                                     "drop-msg": 0,
                                     "sent-msg": 0,
                                     "type": "u"}],
                        "params": "-i u:egress_flow_data_source,u:vportscan_alerts",
                        "path": "/usr/bin/nemea/vportscan_detector",
                        "status": "running"}}