ARK status awk issue with zabbix-agent2

Question

ARK status awk issue with zabbix-agent2

jurim76 opened this issue 4 years ago · comments

Debian default mawk doesn't support zfs.arcstats - "Value of type "string" is not suitable for value type "Numeric (unsigned)". Value "awk: program limit exceeded: maximum number of fields size=32767 FILENAME="/proc/spl/kstat/zfs/arcstats" FNR=48 NR=48"

ARC stats from /proc/spl/kstat/zfs/arcstats
UserParameter=zfs.arcstats[*],awk '/^$1/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats

Gawk has no such limitation, but prints nothing while running "gawk '/^misses/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats" (with two dollar signs)
"gawk '/^misses/ {printf $3;}' /proc/spl/kstat/zfs/arcstats" returns 1021618956 integer as expected (with one dollar sign)

Proxmox 5 with Debian 9 stretch, gawk 4.1.4+dfsg-1, mawk 1.3.3-17+b3

AceSlash · Answer 1 · Mon Aug 17 2020 16:35:09 GMT+0800 (China Standard Time)

This is very strange because this userparameter was never changed and is several years old. It worked fine on Debian 9.

I will have to install a Debian 9 to test it but I suppose the issue is somehow specific to Proxmox since it didn't happen on Debian 9 before.

stefanux · Answer 2 · Mon Aug 24 2020 01:02:05 GMT+0800 (China Standard Time)

this is no proxmox issue, this error exists on vanilla debian 9 (and on debian 10).

actually i take that back, it works perfectly with default mawk. thanks a lot for the template!

AceSlash · Answer 3 · Mon Aug 24 2020 22:49:52 GMT+0800 (China Standard Time)

I don't see the issue on any Debian 10 and I also tested it on Debian 9, it works.

Could you please do this:

uname -a
cat /etc/os-release 
sudo -u zabbix zabbix_agentd -t zfs.arcstats[hits]
sudo -u zabbix zabbix_agentd -t zfs.arcstats[miss]
dpkg -l|grep awk

For reference, this is what I have on one of my last Debian 9:

root@srv-201:~ # uname -a
Linux srv-201 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1 (2018-04-29) x86_64 GNU/Linux
root@srv-201:~ # cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@srv-201:~ # sudo -u zabbix zabbix_agentd -t zfs.arcstats[hits]
zfs.arcstats[hits]                            [t|203642662586]
root@srv-201:~ # sudo -u zabbix zabbix_agentd -t zfs.arcstats[miss]
zfs.arcstats[miss]                            [t|40278493990]
root@srv-201:~ # dpkg -l|grep awk
ii  mawk                                1.3.3-17+b3                             amd64        a pattern scanning and text processing language

And on a Debian 10:

root@srv-328:~ # uname -a
Linux srv-328 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux
root@srv-328:~ # cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@srv-328:~ # sudo -u zabbix zabbix_agentd -t zfs.arcstats[hits]
zfs.arcstats[hits]                            [t|290778]
root@srv-328:~ # sudo -u zabbix zabbix_agentd -t zfs.arcstats[miss]
zfs.arcstats[miss]                            [t|33]
root@srv-328:~ # dpkg -l|grep awk
ii  mawk                           1.3.3-17+b3                  amd64        a pattern scanning and text processing language

I'm monitoring quite a lot of Debian so I'm confident that it works. I would like to reproduce your issue but I don't see how.

AceSlash · Answer 4 · Mon Aug 24 2020 22:59:47 GMT+0800 (China Standard Time)

Actually, could you try to replace awk by the absolute path to mawk?

 UserParameter=zfs.arcstats[*],/usr/bin/mawk '/^$1/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats

Also check your alternative config:

update-alternatives --list awk

The issue may just be that gawk takes over mawk, if that's the case, using mawk directly would fix this, but it may not be possible for bsd systems to use that.

Juri Malinovski · Answer 5 · Wed Aug 26 2020 18:06:53 GMT+0800 (China Standard Time)

@AceSlash take a look at awk print "$$"

lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.13 (stretch)
Release: 9.13
Codename: stretch

update-alternatives --list awk
/usr/bin/mawk

awk print with double "$$"
/usr/bin/mawk '/^mfu_size/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats
mawk: program limit exceeded: maximum number of fields size=32767
FILENAME="/proc/spl/kstat/zfs/arcstats" FNR=54 NR=54

awk '/^mfu_size/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats
mawk: program limit exceeded: maximum number of fields size=32767
FILENAME="/proc/spl/kstat/zfs/arcstats" FNR=54 NR=54
awk print with single "$"
/usr/bin/mawk '/^mfu_size/ {printf $3;}' /proc/spl/kstat/zfs/arcstats
3979592704

awk '/^mfu_size/ {printf $3;}' /proc/spl/kstat/zfs/arcstats
3979592704

AceSlash · Answer 6 · Mon Aug 31 2020 06:36:42 GMT+0800 (China Standard Time)

The double $$ is used by the zabbix agent to actually only use a single $. It's because zabbix uses the single $ for its own macro.

So UserParameter=zfs.arcstats[*],awk '/^$1/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats doesn't actually execute awk with double $$. It does this instead:

awk '/^[ZABBIX MACRO RESULT 1]/ {printf $3;}' /proc/spl/kstat/zfs/arcstats

You will find the relevant documentation here: https://www.zabbix.com/documentation/current/manual/config/items/userparameters#flexible_user_parameters

Juri Malinovski · Answer 7 · Mon Aug 31 2020 18:47:45 GMT+0800 (China Standard Time)

The double $$ is used by the zabbix agent to actually only use a single $. It's because zabbix uses the single $ for its own macro.

Ok, got it. Double $$ is valid for legacy zabbix-agent, but I'm using zabbix-agent2 and double $$ is not required in this case.

AceSlash · Answer 8 · Tue Sep 01 2020 04:53:39 GMT+0800 (China Standard Time)

Okay, that's interesting! I'm still using the legacy agent everywhere. If that's the source of the error, I'll have to create an other file for zabbix-agent2. I will have to install it on at least one server and test it first though.

Florian · Answer 9 · Fri Sep 18 2020 01:22:20 GMT+0800 (China Standard Time)

Just wanted to chime in:
I also have the exact same problem (Debian 10 with agent2) which got solved by removing the double "$" sign.

pwfels · Answer 10 · Wed Jan 06 2021 22:51:43 GMT+0800 (China Standard Time)

Just wanted to chime in:
I also have the exact same problem (Debian 10 with agent2) which got solved by removing the double "$" sign.

same here! Thanks for help!

mjanssens · Answer 11 · Sat Jan 30 2021 21:56:21 GMT+0800 (China Standard Time)

Just to let you know. I corrected the conf file to single $. It was running fine, zabbix-agent2, version 5.0.7
I just upgraded to version 5.0.8 and my arc stats didn't give any data.
So I set it back to $$ and arc stats is working again.

Edit: I changed all $$ back to the state of conf file in this repo, now the vdev state and errors data is working again too.

Juri Malinovski · Answer 12 · Mon Feb 22 2021 18:17:55 GMT+0800 (China Standard Time)

Seems that zabbix team reversed old behaviour with double $$ in recent zabbix-agent2 releases