Not collecting zfs_pool metrics for ZFS >= 2.1

[pando85]

Host operating system: output of uname -a

Linux XXXX 5.15.0-79-generic #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.6.0 (branch: HEAD, revision: ff7f9d69b645cb691dd3e84dc3afc88f5c006962)
  build user:       root@f9c3ed0cfbd3
  build date:       20230527-12:04:27
  go version:       go1.20.4
  platform:         linux/arm64
  tags:             netgo osusergo static_build

node_exporter command line flags

/bin/node_exporter --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root --path.udev.data=/host/root/run/udev/data --web.listen-address=[0.0.0.0]:9100 --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/) --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

node_exporter log output

ts=2023-08-21T22:10:36.488Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.6.0, branch=HEAD, revision=ff7f9d69b645cb6
91dd3e84dc3afc88f5c006962)"                                                                                                                                   
ts=2023-08-21T22:10:36.489Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.20.4, platform=linux/amd64, user=root@f9c3ed0cfb
d3, date=20230527-12:03:54, tags=netgo osusergo static_build)"                                                                                                
ts=2023-08-21T22:10:36.490Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|lo
op|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$                                                                                                                      
ts=2023-08-21T22:10:36.493Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag
=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)                                                                                                   
ts=2023-08-21T22:10:36.493Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(a
utofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinux
fs|squashfs|sysfs|tracefs)$                                                                                                                                   
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:110 level=info msg="Enabled collectors"                                                                   
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=arp                                                                              
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=bcache                                                                           
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=bonding                                                                          
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=btrfs                                                                            
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=conntrack                                                                        
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=cpu                                                                              
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=cpufreq                                                                          
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=diskstats                                                                        
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=dmi                                                                              
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=edac                                                                             
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=entropy                                                                          
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=fibrechannel                                                                     
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=filefd                                                                           
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=filesystem                                                                       
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=hwmon                                                                            
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=infiniband                                                                       
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=ipvs                                                                             
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=loadavg                                                                          
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=mdadm                                                                            
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=meminfo                                                                          
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=netclass                                                                         
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=netdev                                                                           
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=netstat                                                                          
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=nfs                                                                              
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=nfsd                                                                             
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=nvme                                                                             
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=os                                                                               
ts=2023-08-21T22:10:36.494Z caller=node_exporter.go:117 level=info collector=powersupplyclass                                                                 
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=pressure                                                                         
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=rapl                                                                             
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=schedstat                                                                        
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=selinux                                                                          
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=sockstat                                                                         
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=softnet                                                                          
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=stat                                                                             
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=tapestats                                                                        
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=textfile                                                                         
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=thermal_zone                                                                     
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=time                                                                             
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=timex                                                                            
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=udp_queues                                                                       
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=uname                                                                            
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=vmstat                                                                           
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=xfs                                                                              
ts=2023-08-21T22:10:36.495Z caller=node_exporter.go:117 level=info collector=zfs                                                                              
ts=2023-08-21T22:10:36.496Z caller=tls_config.go:274 level=info msg="Listening on" address=:9100                                                              
ts=2023-08-21T22:10:36.497Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=:9100 

Are you running node_exporter in Docker?

Yes, crio with image quay.io/prometheus/node-exporter:v1.6.0

What did you do that produced an error?

I'm collecting ZFS metrics, after upgrading ZFS to 2.1 I miss some node_zfs_zpool_* metrics:

• node_zfs_zpool_nread
• node_zfs_zpool_nwritten
• node_zfs_zpool_rcnt
• node_zfs_zpool_reads
• node_zfs_zpool_rlentime
• node_zfs_zpool_rtime
• node_zfs_zpool_rupdate
• node_zfs_zpool_wcnt
• node_zfs_zpool_wlentime
• node_zfs_zpool_writes
• node_zfs_zpool_wtime
• node_zfs_zpool_wupdate

What did you expect to see?

These metrics:

• node_zfs_zpool_writes
• node_zfs_zpool_nwritten
• node_zfs_zpool_reads
• node_zfs_zpool_nread
• node_zfs_zpool_nunlinks
• node_zfs_zpool_nunlinked

What did you see instead?

The rest of the node_zfs_* metrics are OK.

Interesting related link.

[Johannes 'fish' Ziemke]

Probably duplicate of #2068, otherwise feel free to re-open