Route withdraw missing update nlri message

[enag11]

Exabgp: 4.2.21

Encountered an issue when using two exabgps, will label primary and backup for clarity.
On initial startup if announce a route on the primary then withdraw the route from the backup the nlri information is not sent to the primary and the route is not removed on the primary. Subsequently if the route is announced to the Primary then withdrawn from the Backup the NRLI information is sent and received.

Below are the steps to reproduce.
Announce route on primary:
14:53:51 | 134471 | process | command from process announce-routes : announce flow route {match { source 1.1.1.1/32; destination 2.2.2.2/32;}}
14:53:51 | 134471 | reactor | async | announce-routes | announce flow route {match { source 1.1.1.1/32; destination 2.2.2.2/32;}}
14:53:51 | 134471 | configuration | > route |
14:53:51 | 134471 | configuration | > match |
14:53:51 | 134471 | configuration | . source | '1.1.1.1/32'
14:53:51 | 134471 | configuration | . destination | '2.2.2.2/32'
14:53:51 | 134471 | configuration | < match |
14:53:51 | 134471 | configuration | < route |
14:53:51 | 134471 | api | flow added to neighbor 198.18.18.114 local-ip 198.18.18.170 local-as 100 peer-as 100 router-id 198.18.18.170 family-allowed in-open, neighbor 198.18.18.90 local-ip 198.18.18.170 local-as 100 peer-as 100 router-id 198.18.18.170 family-allowed in-open : flow destination-ipv4 2.2.2.2/32 source-ipv4 1.1.1.1/32
14:53:51 | 134471 | outgoing-1 | sending TCP payload ( 58) FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 003A 0200 0000 2340 0101 0040 0200 4005 0400 0000 6480 0E12 0001 8500 000C 0120 0202 0202 0220 0101 0101
14:53:51 | 134471 | outgoing-1 | >> 1 UPDATE(s)
14:53:51 | 134471 | outgoing-1 | sending TCP payload ( 58) FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 003A 0200 0000 2340 0101 0040 0200 4005 0400 0000 6480 0E12 0001 8500 000C 0120 0202 0202 0220 0101 0101
14:53:51 | 134471 | outgoing-1 | >> 1 UPDATE(s)

Route received on backup
14:53:51 | 130225 | parser | decoded UPDATE ( 0) json { "exabgp": "4.0.1", "time": 1660748031.31912, "host" : "afx-exabgp41", "pid" : 130225, "ppid" : 130224, "counter": 1, "type": "update", "neighbor": { "address": { "local": "198.18.18.114", "peer": "198.18.18.170" }, "asn": { "local": 100, "peer": 100 } , "direction": "in", "message": { "update": { "attribute": { "origin": "igp", "local-preference": 100 }, "announce": { "ipv4 flow": { "no-nexthop": [ { "destination-ipv4": [ "2.2.2.2/32" ], "source-ipv4": [ "1.1.1.1/32" ], "string": "flow destination-ipv4 2.2.2.2/32 source-ipv4 1.1.1.1/32" } ] } } } } } }
14:53:51 | 130225 | peer-1 | << UPDATE #2
14:53:51 | 130225 | peer-1 | UPDATE #2 nlri ( 13) flow destination-ipv4 2.2.2.2/32 source-ipv4 1.1.1.1/32

Withdraw routed from Backup
14:54:09 | 130225 | process | command from process announce-routes : withdraw flow route {match { source 1.1.1.1/32; destination 2.2.2.2/32;}}
14:54:09 | 130225 | reactor | async | announce-routes | withdraw flow route {match { source 1.1.1.1/32; destination 2.2.2.2/32;}}
14:54:09 | 130225 | process | process announce-routes ended, restarting it
14:54:09 | 130225 | process | terminating process announce-routes
14:54:09 | 130225 | process | forked process announce-routes
14:54:09 | 130225 | configuration | > route |
14:54:09 | 130225 | configuration | > match |
14:54:09 | 130225 | configuration | . source | '1.1.1.1/32'
14:54:09 | 130225 | configuration | . destination | '2.2.2.2/32'
14:54:09 | 130225 | configuration | < match |
14:54:09 | 130225 | configuration | < route |
14:54:09 | 130225 | api | flow removed from neighbor 198.18.18.170 local-ip 198.18.18.114 local-as 100 peer-as 100 router-id 198.18.18.114 family-allowed in-open : flow destination-ipv4 2.2.2.2/32 source-ipv4 1.1.1.1/32
14:54:09 | 130225 | incoming-1 | sending TCP payload ( 37) FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 0025 0200 0000 0E40 0101 0040 0200 4005 0400 0000 64
14:54:09 | 130225 | incoming-1 | >> 1 UPDATE(s)

Primary does not received the Update NLRI message
14:54:09 | 134471 | parser | decoded UPDATE ( 0) json { "exabgp": "4.0.1", "time": 1660748049.9560182, "host" : "afx-exabgp4", "pid" : 134471, "ppid" : 134470, "counter": 1, "type": "update", "neighbor": { "address": { "local": "198.18.18.170", "peer": "198.18.18.114" }, "asn": { "local": 100, "peer": 100 } , "direction": "in", "message": { "update": { "attribute": { "origin": "igp", "local-preference": 100 } } } } }
14:54:09 | 134471 | peer-1 | << UPDATE #2

To continue this scenario if the route is readded on the primary then followed by a remove from the backup, the nlri information is sent/received and the route is removed from the primary

Announce route to the Primary
14:54:18 | 134471 | api | flow added to neighbor 198.18.18.114 local-ip 198.18.18.170 local-as 100 peer-as 100 router-id 198.18.18.170 family-allowed in-open, neighbor 198.18.18.90 local-ip 198.18.18.170 local-as 100 peer-as 100 router-id 198.18.18.170 family-allowed in-open : flow destination-ipv4 2.2.2.2/32 source-ipv4 1.1.1.1/32
14:54:18 | 134471 | outgoing-1 | sending TCP payload ( 58) FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 003A 0200 0000 2340 0101 0040 0200 4005 0400 0000 6480 0E12 0001 8500 000C 0120 0202 0202 0220 0101 0101
14:54:18 | 134471 | outgoing-1 | >> 1 UPDATE(s)
14:54:18 | 134471 | outgoing-1 | sending TCP payload ( 58) FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 003A 0200 0000 2340 0101 0040 0200 4005 0400 0000 6480 0E12 0001 8500 000C 0120 0202 0202 0220 0101 0101
14:54:18 | 134471 | outgoing-1 | >> 1 UPDATE(s)

Backup receives the route
14:54:18 | 130225 | parser | decoded UPDATE ( 0) json { "exabgp": "4.0.1", "time": 1660748058.3817239, "host" : "afx-exabgp41", "pid" : 130225, "ppid" : 130224, "counter": 2, "type": "update", "neighbor": { "address": { "local": "198.18.18.114", "peer": "198.18.18.170" }, "asn": { "local": 100, "peer": 100 } , "direction": "in", "message": { "update": { "attribute": { "origin": "igp", "local-preference": 100 }, "announce": { "ipv4 flow": { "no-nexthop": [ { "destination-ipv4": [ "2.2.2.2/32" ], "source-ipv4": [ "1.1.1.1/32" ], "string": "flow destination-ipv4 2.2.2.2/32 source-ipv4 1.1.1.1/32" } ] } } } } } }
14:54:18 | 130225 | peer-1 | << UPDATE #3
14:54:18 | 130225 | peer-1 | UPDATE #3 nlri ( 13) flow destination-ipv4 2.2.2.2/32 source-ipv4 1.1.1.1/32

Now if the route is withdrawn from the backup the nlri message is sent

Withdraw route from Backup
14:54:31 | 130225 | process | command from process announce-routes : withdraw flow route {match { source1.1.1/32; destination 2.2.2.2/32;}}
14:54:31 | 130225 | reactor | async | announce-routes | withdraw flow route {match { source 1.1.1.1/32; tination 2.2.2.2/32;}}
14:54:31 | 130225 | configuration | > route |
14:54:31 | 130225 | configuration | > match |
14:54:31 | 130225 | configuration | . source | '1.1.1.1/32'
14:54:31 | 130225 | configuration | . destination | '2.2.2.2/32'
14:54:31 | 130225 | configuration | < match |
14:54:31 | 130225 | configuration | < route |
14:54:31 | 130225 | api | flow removed from neighbor 198.18.18.170 local-ip 198.18.18.114 local-as 1peer-as 100 router-id 198.18.18.114 family-allowed in-open : flow destination-ipv4 2.2.2.2/32 source-ipv4 1.1./32
14:54:31 | 130225 | incoming-1 | sending TCP payload ( 56) FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 0038 020000 2140 0101 0040 0200 4005 0400 0000 6480 0F10 0001 850C 0120 0202 0202 0220 0101 0101
14:54:31 | 130225 | incoming-1 | >> 1 UPDATE(s)

Route withdraw received on Primary and NLRI information received and route is removed
14:54:31 | 134471 | parser | decoded UPDATE ( 0) json { "exabgp": "4.0.1", "time": 1660748071.9937084, "host" : "afx-exabgp4", "pid" : 134471, "ppid" : 134470, "counter": 2, "type": "update", "neighbor": { "address": { "local": "198.18.18.170", "peer": "198.18.18.114" }, "asn": { "local": 100, "peer": 100 } , "direction": "in", "message": { "update": { "attribute": { "origin": "igp", "local-preference": 100 }, "withdraw": { "ipv4 flow": [ { "destination-ipv4": [ "2.2.2.2/32" ], "source-ipv4": [ "1.1.1.1/32" ], "string": "flow destination-ipv4 2.2.2.2/32 source-ipv4 1.1.1.1/32" } ] } } } } }
14:54:31 | 134471 | peer-1 | << UPDATE #3
14:54:31 | 134471 | peer-1 | UPDATE #3 nlri ( 13) flow destination-ipv4 2.2.2.2/32 source-ipv4 1.1.1.1/32

[thomas-mangin]

ExaBGP will not send a route already in its the ADJ-RIB-OUT, receiving a withdrawal for the same route from the peer does not remove the route from the ADJ-RIB-OUT. Would that explain what you are seeing in your lab?

[thomas-mangin]

If you disable the ADJ-RIB-{IN,OUT} for the peer, the check is removed (not read the code, this is from memory).

[thomas-mangin]

As this option is only visible in the code, I am adding what it should look like here for clarity.

neighbor ... {
   adj-rib-in false;
   adj-rib-out false;
}

[thomas-mangin]

The reason for this behaviour is that ExaBGP has no global RIB (nothing connecting the RIB-IN and RIB-OUT). So even if a valid withdrawal is received from the peer, it will not find its way to the ADJ-RIB-OUT of the peer.

By default, ExaBGP keeps track of routes sent and does not send duplicates. This design decision was made to allow naive API scripts to send the same route over and over without spamming the network with the extra UDPATE messages. They can be therefore be verbose and not have to track state and can also fail and restart without causing all the routes sent to be removed.

[enag11]

Thanks for the response. Please allow me to a few days to work through your responses. I will provide a follow up. thanks