Process brick after sending many announcements
eKrajnak opened this issue · comments
** Bug Report **
Hello, in our setup where ExaBGP announces IPv6 delegated prefixes delegated by Kea DHCP, we found the following bug. The IP addresses and ASN were obfuscated.
Describe the bug
ExaBGP runs with minimal config:
process announce-routes {
run node web_server.js;
encoder json;
}
neighbor 2aff:ffff:ab:1::31 {
router-id 172.18.255.2;
local-address 2aff:ffff:ef:9::93;
local-as 65555;
peer-as 65555;
api {
processes [announce-routes];
}
}
After the web_server.js sends to ExaBGP the following command
announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1
for exactly 13104 times, the ExaBGP process bricks. It does not matter if you send announcements in 2 minutes or 2 weeks. Running process with -d shows that no new command are processed (they are ignored). The connected BGP neighbor will fall to ACTIVE state after hold timer expire.
If I add some other data to annoucement (e.g. community), it has no effect (still needs 13104 annoucements to brick).
If I tried to "split" annoucements like 6552x 2aff:ffff:900::/56 and then 6552x 2aff:ffff:901::/56 it has no effect. The sum 13104 is important. It bricks anyway.
Debug console:
23:50:25 | 3606791 | process | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000]
23:50:25 | 3606791 | reactor | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process | responding to announce-routes : done
23:50:25 | 3606791 | process | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000]
23:50:25 | 3606791 | reactor | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process | responding to announce-routes : done
23:50:25 | 3606791 | process | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000]
23:50:25 | 3606791 | reactor | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process | responding to announce-routes : done
23:50:25 | 3606791 | process | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000]
23:50:25 | 3606791 | reactor | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process | responding to announce-routes : done
23:50:25 | 3606791 | process | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000]
23:50:25 | 3606791 | reactor | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process | responding to announce-routes : done
^C23:51:43 | 3606791 | reactor | ^C received
23:51:43 | 3606791 | reactor | performing shutdown
23:51:43 | 3606791 | outgoing-1 | stop, message [shutting down]
23:51:43 | 3606791 | outgoing-1 | outgoing-1 2aff:ffff:ef:9::93-2aff:ffff:ab:1::31, closing connection
Environment:
- Debian 11, 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1
exabgp/oldstable,now 4.2.8-2 all [installed]
python3-exabgp/oldstable,now 4.2.8-2 all [installed,automatic]
python3/oldstable,now 3.9.2-3 amd64 [installed,automatic]
Additional context
I think the magic constant can be affected by some environment conditions, so please, try to reproduce also with higher number.
If you do not flush the stdout of your application, the data reaches the pipe but not ExaBGP.
23:50:25 | 3606791 | process | command from process announce-routes : announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [65001:1000]
23:50:25 | 3606791 | reactor | async | announce-routes | announce route 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community [ 65001:1000 ]
23:50:25 | 3606791 | configuration | . route | '2aff:ffff:900::/56' 'next-hop' '2aff:ffff:455:1::1' 'community' '[' '65001:1000' ']'
23:50:25 | 3606791 | api | route added to neighbor 2aff:ffff:ab:1::31 local-ip 2aff:ffff:ef:9::93 local-as 65555 peer-as 65555 router-id 172.20.12.2 family-allowed in-open : 2aff:ffff:900::/56 next-hop 2aff:ffff:455:1::1 community 65001:1000
23:50:25 | 3606791 | process | responding to announce-routes : done
ExaBGP does process the route; it must establish that it was already sent to the peer and does nothing.
You may want to try to add this option to your neighbor to see if it helps:
adj-rib-out false;
Also please look at the API section of https://github.com/Exa-Networks/exabgp/wiki/Migration-from-3.4-to-4.0 if you do not wish to see it stop processing data when the pipe from ExaBGP to you app fills.
Oh, I missed that "pipe filling" behaviour. After I switched ACK to false, everything is running correctly. Sorry for disturbing.
Thanks!