1.7.10 print prugin and pre_tag_label_filter failure

Question

1.7.10 print prugin and pre_tag_label_filter failure

danyrlz opened this issue 6 months ago · comments

Hello Paolo,

After upgrading from 1.7.8-git (20211108-1 (7312110)) to 1.7.10-git (20231207-0 (1de6594)) our csv print plugin and pretag filter no longer work(csv file is not generated).
No errors are observed in the debug. Here is the info:

#1.7.8-git(WORKING OK) Config: same as 1.7.10 below without line "print_output_separator: ,"

#PRETAG MAP
#
set_label=ing ip=0.0.0.0/0 direction=0 jeq=src
set_label=egr ip=0.0.0.0/0 direction=1
set_label=other ip=0.0.0.0/0

set_tag=xxxx ip=0.0.0.0/0 src_mac=xx:xx:xx:xx:xx:xx label=src jeq=dst
set_tag=yyyy ip=0.0.0.0/0 src_mac=yy:yy:yy:yy:yy:yy label=src jeq=dst

set_tag2=xxxx ip=0.0.0.0/0 dst_mac=xx:xx:xx:xx:xx:xx label=dst
set_tag2=yyyy ip=0.0.0.0/0 dst_mac=yy:yy:yy:yy:yy:yy label=dst
.....

#1.7.10-git(FAILURE) config:

#
logfile: /var/log/pmacct/sfacctd.log
#
plugins: print[mac2], print[bgp2]

timestamps_secs: true
print_output: csv
print_output_separator: ,
print_output_file[mac2]: /db/csv/sf_mac1.csv
print_output_file[bgp2]: /db/csv/sf_bgp1.csv
print_trigger_exec[bgp2]: /usr/local/scripts/cpimport/csv_rearr_sf_bgp1.sh
print_trigger_exec[mac2]: /usr/local/scripts/cpimport/csv_rearr_sf_mac1.sh
print_cache_entries: 5000000

sql_table_version: 9
sql_locking_style: none
sql_table_type: bgp
sql_optimize_clauses: true
sql_dont_try_update: true
sql_multi_values: 409680000
sql_cache_entries: 5000000


print_history[mac2]: 5m
print_history_roundoff[mac2]: m
print_refresh_time[mac2]: 300

print_history[bgp2]: 5m
print_history_roundoff[bgp2]: m
print_refresh_time[bgp2]: 300

sfacctd_port: 6344
sfacctd_renormalize: true
networks_cache_entries: 1000003
bgp_daemon: true
bgp_daemon_ip: x.x.x.x
bgp_daemon_max_peers: 100
#
sfacctd_as: bgp
bgp_src_as_path_type: bgp
bgp_peer_src_as_type: bgp
bgp_agent_map: /etc/pmacct/agent_to_peer5.map
pre_tag_map: /etc/pmacct/pretag/peer5temp.map
pre_tag_label_filter[bgp2]: ing
pre_tag_label_filter[mac2]: -null
#
#
bgp_daemon_pipe_size: 10240000
plugin_buffer_size: 102400
plugin_pipe_size:  102400000

aggregate[bgp2]: peer_src_ip, src_mac, dst_mac, vlan, src_host, dst_host, src_as, dst_as, peer_src_as, peer_dst_as, as_path, tag, tag2
aggregate[mac2]: peer_src_ip, src_mac, dst_mac, vlan, src_as, dst_as, peer_src_as, peer_dst_as, as_path, tag, tag2

#PRETAG MAP
#
set_label=ing    ip=0.0.0.0/0  direction=0  label=ing    jeq=src
set_label=egr    ip=0.0.0.0/0  direction=1  label=egr   
set_label=other  ip=0.0.0.0/0
#
set_tag=xxxx ip=0.0.0.0/0 src_mac=xx:xx:xx:xx:xx:xx label=src jeq=dst
set_tag=yyyy ip=0.0.0.0/0 src_mac=yy:yy:yy:yy:yy:yy label=src jeq=dst

set_tag2=xxxx ip=0.0.0.0/0 dst_mac=xx:xx:xx:xx:xx:xx label=dst
set_tag2=yyyy ip=0.0.0.0/0 dst_mac=yy:yy:yy:yy:yy:yy label=dst

RESULT:

2023-12-13T13:27:58Z INFO ( default/core ): sFlow Accounting Daemon, sfacctd 1.7.10-git (20231207-0 (1de6594))
2023-12-13T13:27:58Z INFO ( default/core ): '--enable-mysql' '--enable-debug' '--enable-jansson' '--enable-kafka' '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'
2023-12-13T13:27:58Z INFO ( default/core ): Reading configuration file '/etc/pmacct/sfacct5gtemp.conf'.
2023-12-13T13:27:58Z INFO ( default/core ): [/etc/pmacct/agent_to_peer5.map] (re)loading map.
2023-12-13T13:27:58Z INFO ( default/core ): [/etc/pmacct/agent_to_peer5.map] map successfully (re)loaded.
2023-12-13T13:27:58Z INFO ( default/core/BGP ): maximum BGP peers allowed: 100
2023-12-13T13:27:58Z INFO ( default/core/BGP ): bgp_daemon_pipe_size: obtained=425984 target=10240000.
2023-12-13T13:27:58Z INFO ( default/core/BGP ): waiting for BGP data on interface=all ip=z.z.z.z port=179/tcp

2023-12-13T13:28:03Z INFO ( mac2/print ): plugin_pipe_size=102400000 bytes plugin_buffer_size=102400 bytes
2023-12-13T13:28:03Z INFO ( mac2/print ): ctrl channel: obtained=212992 bytes target=8000 bytes
2023-12-13T13:28:03Z INFO ( bgp2/print ): plugin_pipe_size=102400000 bytes plugin_buffer_size=102400 bytes
2023-12-13T13:28:03Z INFO ( bgp2/print ): ctrl channel: obtained=212992 bytes target=8000 bytes
2023-12-13T13:28:03Z INFO ( mac2/print ): cache entries=5000000 base cache memory=17160000000 bytes
2023-12-13T13:28:03Z INFO ( default/core ): [/etc/pmacct/pretag/peer5temp.map] (re)loading map.
2023-12-13T13:28:03Z INFO ( bgp2/print ): cache entries=5000000 base cache memory=17160000000 bytes
2023-12-13T13:28:03Z INFO ( default/core ): [/etc/pmacct/pretag/peer5temp.map] map successfully (re)loaded.
2023-12-13T13:28:03Z INFO ( default/core ): [/etc/pmacct/pretag/peer5temp.map] (re)loading map.
2023-12-13T13:28:03Z INFO ( default/core ): [/etc/pmacct/pretag/peer5temp.map] map successfully (re)loaded.
2023-12-13T13:28:03Z INFO ( default/core ): [/etc/pmacct/pretag/peer5temp.map] (re)loading map.
2023-12-13T13:28:03Z INFO ( default/core ): [/etc/pmacct/pretag/peer5temp.map] map successfully (re)loaded.
2023-12-13T13:28:03Z INFO ( default/core ): waiting for sFlow data on interface=all ip=:: port=6344/udp

2023-12-13T13:28:11Z INFO ( default/core/BGP ): [z.z.z.z] BGP peers usage: 1/100
2023-12-13T13:28:11Z INFO ( default/core/BGP ): [z.z.z.z] Capability: MultiProtocol [1] AFI [1] SAFI [1]
2023-12-13T13:28:11Z INFO ( default/core/BGP ): [z.z.z.z] Capability: 4-bytes AS [65] ASN [zzzzz]
2023-12-13T13:28:11Z INFO ( default/core/BGP ): [z.z.z.z] BGP_OPEN: Local AS: xzxzxz Remote AS: zzzzz HoldTime: 240

2023-12-13T13:30:01Z INFO ( mac2/print ): *** Purging cache - START (PID: 68282) ***
2023-12-13T13:30:01Z INFO ( mac2/print ): *** Purging cache - END (PID: 68282, QN: 85171/85171, ET: 0) ***
2023-12-13T13:30:01Z INFO ( bgp2/print ): *** Purging cache - START (PID: 68293) ***
2023-12-13T13:30:01Z INFO ( bgp2/print ): *** Purging cache - END (PID: 68293, QN: 0/0, ET: X) ***

2023-12-13T13:35:01Z INFO ( mac2/print ): *** Purging cache - START (PID: 68363) ***
2023-12-13T13:35:01Z INFO ( mac2/print ): *** Purging cache - END (PID: 68363, QN: 129574/129574, ET: 0) ***
2023-12-13T13:35:01Z INFO ( bgp2/print ): *** Purging cache - START (PID: 68374) ***
2023-12-13T13:35:01Z INFO ( bgp2/print ): *** Purging cache - END (PID: 68374, QN: 0/0, ET: X) ***

Paolo Lucente · Answer 1 · Thu Dec 14 2023 04:50:54 GMT+0800 (China Standard Time)

Hi @danyrlz ,

Thanks for reporting this!

Can you please remove one moment the pre_tag_label_filter to observe what happens? Is all traffic labelled as other? Or what happens? It may also help if you could send me a sample of a sFlow packet, especially to test the direction part of matching / non-matching.

Paolo

Jordan · Answer 2 · Thu Dec 14 2023 15:31:44 GMT+0800 (China Standard Time)

Hello,
it looks like there is more fundamental issue as we get the same without the "direction" key:

.....
pre_tag_label_filter[bgp2]: ing
.....
set_label=ing    ip=0.0.0.0/0   label=ing    jeq=src
#
# START_SRC-Static
set_tag=xxxx ip=0.0.0.0/0 src_mac=xx:xx:xx:xx:xx:xx label=src jeq=dst
# START_DST-Static
set_tag2=xxxx ip=0.0.0.0/0 dst_mac=yy:yy:yy:yy:yy:yy label=dst

RESULT:

2023-12-14T07:25:01Z INFO ( bgp2/print ): *** Purging cache - START (PID: 82609) ***
2023-12-14T07:25:01Z INFO ( bgp2/print ): *** Purging cache - END (PID: 82609, QN: 0/0, ET: X) ***

Here is with disabled filter:

.....
pre_tag_label_filter[mac2]: -null
#pre_tag_label_filter[bgp2]: ing
.....
set_label=ing    ip=0.0.0.0/0   label=ing    jeq=src
#
# START_SRC-Static
set_tag=xxxx ip=0.0.0.0/0 src_mac=xx:xx:xx:xx:xx:xx label=src jeq=dst
# START_DST-Static
set_tag2=xxxx ip=0.0.0.0/0 dst_mac=yy:yy:yy:yy:yy:yy label=dst

RESULT:

2023-12-14T07:30:01Z INFO ( mac2/print ): *** Purging cache - START (PID: 82721) ***
2023-12-14T07:30:01Z INFO ( bgp2/print ): *** Purging cache - START (PID: 82722) ***
2023-12-14T07:30:01Z INFO ( mac2/print ): *** Purging cache - END (PID: 82721, QN: 6779/6779, ET: 0) ***
2023-12-14T07:30:03Z INFO ( bgp2/print ): *** Purging cache - END (PID: 82722, QN: 634790/634790, ET: 2) ***

Paolo Lucente · Answer 3 · Fri Dec 15 2023 04:35:59 GMT+0800 (China Standard Time)

Hi Jordan ( @danyrlz ),

I should have been able to reproduce the issue and fix it. Can you give a try to latest code and confirm whether it works for you?

Paolo

Jordan · Answer 4 · Fri Dec 15 2023 19:47:04 GMT+0800 (China Standard Time)

Hello again,

It seems to work now but we get another error with the same config from 1.7.8:

2023-12-15T11:30:46Z INFO ( default/core ): [/etc/pmacct/pretag/peer5b.map] (re)loading map.
2023-12-15T11:30:46Z INFO ( bgp2/print ): cache entries=5000000 base cache memory=17160000000 bytes
2023-12-15T11:30:46Z INFO ( default/core ): [/etc/pmacct/pretag/peer5b.map] map successfully (re)loaded.
2023-12-15T11:30:46Z INFO ( default/core ): [/etc/pmacct/pretag/peer5b.map] (re)loading map.
2023-12-15T11:30:46Z WARN ( default/core ): [/etc/pmacct/pretag/peer5b.map] Unresolved label 'dst'. Ignoring it.
2023-12-15T11:30:46Z WARN ( default/core ): [/etc/pmacct/pretag/peer5b.map] Unresolved label 'dst'. Ignoring it.
2023-12-15T11:30:46Z WARN ( default/core ): [/etc/pmacct/pretag/peer5b.map] Unresolved label 'dst'. Ignoring it.
2023-12-15T11:30:46Z WARN ( default/core ): [/etc/pmacct/pretag/peer5b.map] Unresolved label 'dst'. Ignoring it.
2023-12-15T11:30:46Z WARN ( default/core ): [/etc/pmacct/pretag/peer5b.map] Unresolved label 'dst'. Ignoring it.
2023-12-15T11:30:46Z WARN ( default/core ): [/etc/pmacct/pretag/peer5b.map] Unresolved label 'dst'. Ignoring it.
2023-12-15T11:30:46Z WARN ( default/core ): [/etc/pmacct/pretag/peer5b.map] Unresolved label 'dst'. Ignoring it.
2023-12-15T11:30:46Z WARN ( default/core ): [/etc/pmacct/pretag/peer5b.map] Unresolved label 'dst'. Ignoring it.
2023-12-15T11:30:46Z WARN ( default/core ): [/etc/pmacct/pretag/peer5b.map] Unresolved label 'dst'. Ignoring it.

We use the same pretag map structure like above. When we use just few lines in the new version it looks ok but when we load the full map(around 500 lines) we get the error.
Is there any limitation or anything new specific here?

Jordan · Answer 5 · Mon Dec 18 2023 23:39:12 GMT+0800 (China Standard Time)

Hello Paolo @paololucente,

It looks like when we have more than 52 lines with "jeq=dst" statement the problem above is present.

Paolo Lucente · Answer 6 · Tue Dec 19 2023 01:45:04 GMT+0800 (China Standard Time)

Hi Jordan @danyrlz , sorry the silence, i was working on some code -- having been able to reproduce your issue. I have some code ready & testing at the moment. I should have news by the end of the day. Paolo

Paolo Lucente · Answer 7 · Tue Dec 19 2023 05:25:53 GMT+0800 (China Standard Time)

Hi Jordan @danyrlz , the last commit ( 547e241 ) should address your remaining issue -- can you give it a try and let me know if all looks to work as expected?

Paolo