[Bug]: apcupsd module is unable to connect to UPS (docker)

Question

[Bug]: apcupsd module is unable to connect to UPS (docker)

v3nko opened this issue a month ago · comments

Bug description

I'm using Netdata in docker container with compose config:

services:
  netdata:
    image: netdata/netdata:stable
    container_name: netdata
...
    environment:
...
      - NETDATA_EXTRA_DEB_PACKAGES=apcupsd
    extra_hosts:
      - "host.docker.internal:host-gateway"
...

and charts.d/apcupsd.conf:

root@venko:/etc/netdata/charts.d# cat apcupsd.conf
declare -A apcupsd_sources=(
    ["main"]="host.docker.internal:3551"
)

The UPS section does not appear, so I've used debug mode in container:

root@venko:/usr/libexec/netdata/plugins.d# ./charts.d.plugin debug 1 apcupsd
...
time=2024-05-08T10:57:27.688+00:00 comm=charts.d.plugin source=collector level=debug tid=2427 thread=charts.d.plugin msg="apcupsd: loading module: './../charts.d/apcupsd.chart.sh'"
time=2024-05-08T10:57:27.692+00:00 comm=charts.d.plugin source=collector level=debug tid=2428 thread=charts.d.plugin msg="apcupsd: loading module configuration: '/usr/lib/netdata/conf.d/charts.d/apcupsd.conf'"
time=2024-05-08T10:57:27.696+00:00 comm=charts.d.plugin source=collector level=debug tid=2429 thread=charts.d.plugin msg="apcupsd: loading module configuration: '/etc/netdata/charts.d/apcupsd.conf'"
2024-05-08 10:57:27: charts.d.plugin: : apcupsd: command 'apcaccess status 127.0.0.1:3551 ' failed with code 1:
 --- BEGIN TRACE ---
Error contacting apcupsd @ 127.0.0.1:3551: Connection refused
 --- END TRACE ---
time=2024-05-08T10:57:27.707+00:00 comm=charts.d.plugin source=collector level=error tid=2437 thread=charts.d.plugin msg="apcupsd: cannot get information for apcupsd server local on 127.0.0.1:3551."
2024-05-08 10:57:27: charts.d.plugin: : apcupsd: command 'apcaccess status $'(\r [main]=host.docker.internal:3551\r)\r' ' failed with code 1:
 --- BEGIN TRACE ---
 [main]=host.docker.internal:3551: No such device or address
 --- END TRACE ---
time=2024-05-08T10:57:27.716+00:00 comm=charts.d.plugin source=collector level=error tid=2443 thread=charts.d.plugin msg="apcupsd: cannot get information for apcupsd server 0 on (\u000D [main]=host.docker.internal:3551\u000D)\u000D."
time=2024-05-08T10:57:27.719+00:00 comm=charts.d.plugin source=collector level=error tid=2444 thread=charts.d.plugin msg="apcupsd: No APC UPSes found available."
time=2024-05-08T10:57:27.722+00:00 comm=charts.d.plugin source=collector level=error tid=2445 thread=charts.d.plugin msg="apcupsd: module's 'apcupsd' check() function reports failure."
...

And it looks like issue somewhere in shell script as expression apcupsd: command 'apcaccess status $'(\r [main]=host.docker.internal:3551\r)\r' does not look right to me.

To ensure that I can actually connect to host's apcupsd I've tried to do this directly:

root@venko:/usr/libexec/netdata/plugins.d# apcaccess status host.docker.internal:3551
APC      : 001,036,0860
DATE     : 2024-05-08 13:58:09 +0300
HOSTNAME : cubi2
VERSION  : 3.14.14 (31 May 2016) debian
UPSNAME  : cubi2
CABLE    : USB Cable
DRIVER   : USB UPS Driver
UPSMODE  : Stand Alone
STARTTIME: 2024-05-08 12:34:31 +0300
MODEL    : Back-UPS RS 900G
STATUS   : ONLINE
LINEV    : 233.0 Volts
LOADPCT  : 3.0 Percent
BCHARGE  : 61.0 Percent
TIMELEFT : 151.1 Minutes
MBATTCHG : 29 Percent
MINTIMEL : 3 Minutes
MAXTIME  : 0 Seconds
SENSE    : Medium
LOTRANS  : 176.0 Volts
HITRANS  : 294.0 Volts
ALARMDEL : No alarm
BATTV    : 26.6 Volts
LASTXFER : No transfers since turnon
NUMXFERS : 0
TONBATT  : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STATFLAG : 0x05000008
SERIALNO : ***
BATTDATE : 2021-05-20
NOMINV   : 230 Volts
NOMBATTV : 24.0 Volts
NOMPOWER : 540 Watts
FIRMWARE : 879.L4 .I USB FW:L4
END APC  : 2024-05-08 13:58:22 +0300

Expected behavior

apcupsd module successfully scrapes UPS data

Steps to reproduce

Install & configure apcupsd on host machine
Deploy Netdata in docker container with NETDATA_EXTRA_DEB_PACKAGES=apcupsd env var, host gateway mapping and charts.d/apcupsd.conf config
Run the container

Installation method

docker

System info

Linux cubi2 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
/etc/lsb-release:DISTRIB_ID=Ubuntu
/etc/lsb-release:DISTRIB_RELEASE=22.04
/etc/lsb-release:DISTRIB_CODENAME=jammy
/etc/lsb-release:DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"
/etc/os-release:PRETTY_NAME="Ubuntu 22.04.4 LTS"
/etc/os-release:NAME="Ubuntu"
/etc/os-release:VERSION_ID="22.04"
/etc/os-release:VERSION="22.04.4 LTS (Jammy Jellyfish)"
/etc/os-release:VERSION_CODENAME=jammy
/etc/os-release:ID=ubuntu
/etc/os-release:ID_LIKE=debian
/etc/os-release:UBUNTU_CODENAME=jammy

Netdata build info

Packaging:
    Netdata Version ____________________________________________ : v1.45.3
    Installation Type __________________________________________ : oci
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ : unknown
    Configure Options __________________________________________ : dummy-configure-command
Default Directories:
    User Configurations ________________________________________ : /etc/netdata
    Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
    Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
    Permanent Databases ________________________________________ : /var/lib/netdata
    Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /usr/share/netdata/web
    Log Files __________________________________________________ : /var/log/netdata
    Lock Files _________________________________________________ : /var/lib/netdata/lock
    Home _______________________________________________________ : /var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 5.15.0-105-generic
    Operating System ___________________________________________ : Ubuntu
    Operating System ID ________________________________________ : ubuntu
    Operating System ID Like ___________________________________ : debian
    Operating System Version ___________________________________ : 22.04.4 LTS (Jammy Jellyfish)
    Operating System Version ID ________________________________ : 12
    Detection __________________________________________________ : /host/etc/os-release
Hardware:
    CPU Cores __________________________________________________ : 4
    CPU Frequency ______________________________________________ : 2500000000
    RAM Bytes __________________________________________________ : 8199729152
    Disk Capacity ______________________________________________ : 2480502915072
    CPU Architecture ___________________________________________ : x86_64
    Virtualization Technology __________________________________ : none
    Virtualization Detection ___________________________________ : none
Container:
    Container __________________________________________________ : docker
    Container Detection ________________________________________ : dockerenv
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : Debian GNU/Linux
    Container Operating System ID ______________________________ : debian
    Container Operating System ID Like _________________________ : unknown
    Container Operating System Version _________________________ : 12 (bookworm)
    Container Operating System Version ID ______________________ : 12
    Container Operating System Detection _______________________ : /etc/os-release
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Back-filling (of higher database tiers) ____________________ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
Database Engines:
    dbengine ___________________________________________________ : YES
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    h2o (web server) ___________________________________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    ZSTD (fast, lossless compression algorithm) ________________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Brotli (generic-purpose lossless compression algorithm) ____ : NO
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libyaml (library for parsing and emitting YAML) ____________ : YES
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : NO
    ebpf (monitor system calls) ________________________________ : NO
    freeipmi (monitor enterprise server H/W) ___________________ : YES
    nfacct (gather netfilter accounting) _______________________ : NO
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : NO
    Xen VBD Error Tracking _____________________________________ : NO
    Logs Management ____________________________________________ : YES
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : YES
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO

Additional info

No response

Ilya Mashchenko · Answer 1 · Wed May 08 2024 22:18:37 GMT+0800 (China Standard Time)

Hi. I can't reproduce the issue.

root@78df9567fddd:/usr/libexec/netdata/plugins.d# cat /etc/netdata/charts.d/apcupsd.conf
declare -A apcupsd_sources=(
    ["main"]="host.docker.internal:3551"
)

root@78df9567fddd:/usr/libexec/netdata/plugins.d# ./charts.d.plugin apcupsd 2>&1 | grep apcupsd
time=2024-05-08T14:18:09.786+00:00 comm=charts.d.plugin source=collector level=info tid=1788 thread=charts.d.plugin msg="main: started from './charts.d.plugin' with options: apcupsd"
time=2024-05-08T14:18:09.816+00:00 comm=charts.d.plugin source=collector level=debug tid=1808 thread=charts.d.plugin msg="apcupsd: is enabled for auto-detection."
time=2024-05-08T14:18:09.861+00:00 comm=charts.d.plugin source=collector level=debug tid=1861 thread=charts.d.plugin msg="main: enabled charts:  ap apcupsd libreswan opensips"
time=2024-05-08T14:18:09.884+00:00 comm=charts.d.plugin source=collector level=debug tid=1869 thread=charts.d.plugin msg="apcupsd: loading module: './../charts.d/apcupsd.chart.sh'"
time=2024-05-08T14:18:09.888+00:00 comm=charts.d.plugin source=collector level=debug tid=1870 thread=charts.d.plugin msg="apcupsd: loading module configuration: '/usr/lib/netdata/conf.d/charts.d/apcupsd.conf'"
time=2024-05-08T14:18:09.891+00:00 comm=charts.d.plugin source=collector level=debug tid=1871 thread=charts.d.plugin msg="apcupsd: loading module configuration: '/etc/netdata/charts.d/apcupsd.conf'"
2024-05-08 14:18:09: charts.d.plugin: : apcupsd: command 'apcaccess status host.docker.internal:3551 ' failed with code 1:
Error contacting apcupsd @ host.docker.internal:3551: Connection refused
time=2024-05-08T14:18:09.902+00:00 comm=charts.d.plugin source=collector level=error tid=1879 thread=charts.d.plugin msg="apcupsd: cannot get information for apcupsd server main on host.docker.internal:3551."
time=2024-05-08T14:18:09.905+00:00 comm=charts.d.plugin source=collector level=error tid=1880 thread=charts.d.plugin msg="apcupsd: No APC UPSes found available."
time=2024-05-08T14:18:09.908+00:00 comm=charts.d.plugin source=collector level=error tid=1881 thread=charts.d.plugin msg="apcupsd: module's 'apcupsd' check() function reports failure."
time=2024-05-08T14:18:09.954+00:00 comm=charts.d.plugin source=collector level=debug tid=1898 thread=charts.d.plugin msg="main: requested to run only for: 'apcupsd'"

Victor K · Answer 2 · Wed May 08 2024 22:53:49 GMT+0800 (China Standard Time)

Thanks for looking into it. Looks even weirder than before. I've tried on a different machine with the same Netdata config and got the expected result like you.

I assumed that it might be file encoding, but in fact, the root cause was in line break characters.

The problematic config file is using CRLF line breaks, as I created it on a Windows machine and deployed using Ansible. LF line breaks work well with file that I created directly on machine with Netdata.

Should I convert/update this issue or create a new one with suggestion to support CRLF line breaks in apcupsd.conf?

Ilya Mashchenko · Answer 3 · Wed May 08 2024 23:16:32 GMT+0800 (China Standard Time)

@v3nko Thank you for finding and solving the problem yourself! No, we won't add CRLF support. apcupsd is a bash collector and won't receive updates other than bug fixes. I plan to rewrite it in go when I get a chance.