Performance data writer causes runaway-series-cardinality in InfluxDB

[Dominik Riva]

Expected Behavior

Performance data writer is "setup and forget" and provides a clear benefit.

Current Behavior

• In large and highly dynamic vCenters (virtual desktop infrastructure) the writer can result in runaway-series-cardinality:

Sep 15 12:02:11 icingalp06 influxd-systemd-start.sh[6778]: ts=2023-09-15T10:02:11.985985Z lvl=warn msg="max-values-per-tag limit may be exceeded soon" log_id=0kFnS2vG000 service=store perc=100% n=100000 max=100000 db_instance=icinga_vspheredb_ictvdivclp01 measurement=VmMemory tag=vm_uuid

[12:03:35 root@icingalp06 ~]$ journalctl -xeu influxdb | grep max-values-per-tag | cut -d' ' -f 19 | sort -u
db_instance=icinga_vspheredb_ictvdivclp01

[12:04:08 root@icingalp06 ~]$ journalctl -xeu influxdb | grep max-values-per-tag | cut -d' ' -f 20 | sort -u
measurement=VirtualDisk
measurement=VirtualNetworkAdapter
measurement=VmCpu
measurement=VmMemory

[12:04:15 root@icingalp06 ~]$ journalctl -xeu influxdb | grep max-values-per-tag | cut -d' ' -f 21 | sort -u
tag=instance
tag=vm_moref
tag=vm_uuid

https://www.influxdata.com/blog/solving-runaway-series-cardinality-when-using-influxdb/

• Also in the module itself, there isn't a clear benefit for setting all this up. Including "fixing" the Grafana dashboards.

So for the time being, we disabled the feature and droppend the InfluxDB icinga_vspheredb_* instances.
Because this feature endangers the Grafana graphs in the monitoring module and I suspect nobody except me ever opened the Grafana dashboard belonging to this module.

Possible Solution

• Performance data writer toggle per vCenter
• Performance data writer sends data in a way to not trigger runaway-series-cardinality
• Add Grafana graphs to the module and not just display the current state of CPU, RAM and the like.

Your Environment

• VMware vCenter®/ESXi™-Version: vCenter Server 6.7.0
• Version/GIT-Hash of this module: 1.6.0
• Icinga Web 2 version: 2.11.4
• Operating System and version: AlmaLinux8
• Webserver, PHP versions: PHP 8.0.30