Process brick after sending many announcements

Question

Process brick after sending many announcements

eKrajnak opened this issue a year ago · comments

** Bug Report **

Hello, in our setup where ExaBGP announces IPv6 delegated prefixes delegated by Kea DHCP, we found the following bug. The IP addresses and ASN were obfuscated.

Describe the bug

ExaBGP runs with minimal config:

process announce-routes {
    run node web_server.js;
    encoder json;
}

neighbor 2aff:ffff:ab:1::31 {
    router-id 172.18.255.2;
    local-address 2aff:ffff:ef:9::93;
    local-as 65555;
    peer-as 65555;

    api {
            processes [announce-routes];
    }
}

After the web_server.js sends to ExaBGP the following command

announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1

for exactly 13104 times, the ExaBGP process bricks. It does not matter if you send announcements in 2 minutes or 2 weeks. Running process with -d shows that no new command are processed (they are ignored). The connected BGP neighbor will fall to ACTIVE state after hold timer expire.

If I add some other data to annoucement (e.g. community), it has no effect (still needs 13104 annoucements to brick).
If I tried to "split" annoucements like 6552x 2aff:ffff:900::/56 and then 6552x 2aff:ffff:901::/56 it has no effect. The sum 13104 is important. It bricks anyway.

Debug console:

23:50:25 | 3606791 | process       | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000] 
23:50:25 | 3606791 | reactor       | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route            | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api           | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process       | responding to announce-routes : done
23:50:25 | 3606791 | process       | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000] 
23:50:25 | 3606791 | reactor       | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route            | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api           | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process       | responding to announce-routes : done
23:50:25 | 3606791 | process       | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000] 
23:50:25 | 3606791 | reactor       | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route            | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api           | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process       | responding to announce-routes : done
23:50:25 | 3606791 | process       | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000] 
23:50:25 | 3606791 | reactor       | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route            | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api           | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process       | responding to announce-routes : done
23:50:25 | 3606791 | process       | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000] 
23:50:25 | 3606791 | reactor       | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route            | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api           | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process       | responding to announce-routes : done
^C23:51:43 | 3606791 | reactor       | ^C received
23:51:43 | 3606791 | reactor       | performing shutdown
23:51:43 | 3606791 | outgoing-1    | stop, message [shutting down]
23:51:43 | 3606791 | outgoing-1    | outgoing-1 2aff:ffff:ef:9::93-2aff:ffff:ab:1::31, closing connection

Environment:

Debian 11, 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1
exabgp/oldstable,now 4.2.8-2 all [installed]
python3-exabgp/oldstable,now 4.2.8-2 all [installed,automatic]
python3/oldstable,now 3.9.2-3 amd64 [installed,automatic]

Additional context

I think the magic constant can be affected by some environment conditions, so please, try to reproduce also with higher number.

Thomas Mangin · Answer 1 · Tue Jul 18 2023 16:24:02 GMT+0800 (China Standard Time)

If you do not flush the stdout of your application, the data reaches the pipe but not ExaBGP.

Thomas Mangin · Answer 2 · Tue Jul 18 2023 16:26:47 GMT+0800 (China Standard Time)

23:50:25 | 3606791 | process       | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000] 
23:50:25 | 3606791 | reactor       | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route            | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api           | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process       | responding to announce-routes : done

ExaBGP does process the route; it must establish that it was already sent to the peer and does nothing.

You may want to try to add this option to your neighbor to see if it helps:

adj-rib-out false;

Thomas Mangin · Answer 3 · Tue Jul 18 2023 16:28:33 GMT+0800 (China Standard Time)

Also please look at the API section of https://github.com/Exa-Networks/exabgp/wiki/Migration-from-3.4-to-4.0 if you do not wish to see it stop processing data when the pipe from ExaBGP to you app fills.

Blažej Krajňák · Answer 4 · Tue Jul 18 2023 18:18:48 GMT+0800 (China Standard Time)

Oh, I missed that "pipe filling" behaviour. After I switched ACK to false, everything is running correctly. Sorry for disturbing.
Thanks!