Zebra overwrites the nexthop ID for a static route

[christophefontaine]

Description

With FRR connected, if a user adds a static route from grcli, Zebra re-creates that same nexthop with a different ID:

grout# add nexthop address 192.168.210.3 iface cv0
grout# show nexthop
VRF  ID  TYPE  FAMILY  IP                         MAC                IFACE  STATE      FLAGS                      ORIGIN
0        L3    IPv6    fe80::b483:51ff:fe05:43ca  b4:83:51:05:43:ca  cv0    reachable  static local gateway link  link
0        L3    IPv6    fe80::b483:51ff:fe05:43cb  b4:83:51:05:43:cb  cv1    reachable  static local gateway link  link
0        L3    IPv4    192.168.210.1              b4:83:51:05:43:ca  cv0    reachable  static local gateway link  link
0    87  L3    IPv4    192.168.210.254            ?                  cv0    new                                   zebra
0    2   L3    IPv4    192.168.210.2              ?                  cv0    new        gateway                    zebra
0    1   L3    IPv4    192.168.210.3              ?                  cv0    new                                   user
0    89  L3    IPv4    192.168.210.6              ?                  cv0    new        gateway                    zebra
grout# add ip route 192.168.1.11/32 via id 1
grout# show ip route
VRF  DESTINATION       NEXT_HOP         ORIGIN        ID  NEXT_HOP_VRF
0    192.168.1.10/32   192.168.210.254  user          87  0
0    192.168.1.11/32   192.168.210.3    user          90  0
0    192.168.1.30/32   192.168.210.6    zebra_static  89  0
0    192.168.210.0/24  192.168.210.1    link              0
grout# show nexthop
VRF  ID  TYPE  FAMILY  IP                         MAC                IFACE  STATE      FLAGS                      ORIGIN
0        L3    IPv6    fe80::b483:51ff:fe05:43ca  b4:83:51:05:43:ca  cv0    reachable  static local gateway link  link
0        L3    IPv6    fe80::b483:51ff:fe05:43cb  b4:83:51:05:43:cb  cv1    reachable  static local gateway link  link
0        L3    IPv4    192.168.210.1              b4:83:51:05:43:ca  cv0    reachable  static local gateway link  link
0    87  L3    IPv4    192.168.210.254            ?                  cv0    new                                   zebra
0    2   L3    IPv4    192.168.210.2              ?                  cv0    new        gateway                    zebra
0    90  L3    IPv4    192.168.210.3              ?                  cv0    new                                   zebra
0    89  L3    IPv4    192.168.210.6              ?                  cv0    new        gateway                    zebra

In vtysh, we see both next hops though:

compute05.mgmt.foobar.space# show nexthop-group rib
ID: 1 (kernel)
     RefCnt: 1 Time to Deletion: 00:00:02
     Uptime: 00:03:18
     VRF: default(IPv4)
     Nexthop Count: 1
     Valid, Installed
     Interface Index: 257
           via 192.168.210.3, cv0 (vrf default), weight 1
ID: 90 (zebra)
     RefCnt: 2
     Uptime: 00:02:58
     VRF: default(IPv4)
     Nexthop Count: 1
     Valid, Installed
     Interface Index: 257
           via 192.168.210.3, cv0 (vrf default), weight 1

Version

grout v0.11.0-62-gc029279c-dirty (DPDK 24.11.1)

[rjarry]

After investigation, this behavior is expected given grout's architecture.

Unlike the Linux kernel which separates nexthop objects (purely ID-based, no dedup) from the neighbor table (ARP/NDP resolution, keyed by address+device), grout merges both into a single L3 nexthop object. The L3 nexthop hash enforces uniqueness by (af, vrf_id, addr) because each object carries ARP state (MAC, probes, held packets). Having two objects for the same address would mean duplicate, potentially inconsistent ARP resolution.

When zebra receives a kernel nexthop (id=1) and a route referencing it, zebra's route resolution creates its own NHG (id=90) for the same gateway. Zebra then installs NHG id=90 via the dplane plugin. In grout, nexthop_lookup() finds the existing nexthop by content (same address), and nexthop_update() overwrites its ID and origin. This is the only possible outcome since grout cannot create a second L3 nexthop with the same address.

This should be harmless in practice:

• Existing routes hold pointers to the nexthop object, not IDs, so they remain valid after the ID/origin change.
• When zebra later tries to delete the old kernel NHG (id=1), the delete is a no-op since id=1 no longer exists and the request uses missing_ok=true.
• Zebra does not re-install kernel-originated routes back through the dplane (filtered by origin), so no route will reference a stale ID.

[rjarry]

For reference, here is the FRR code path (based on frr-10.5.0):

Zebra creates its own NHG copy during route processing:

When rib_process() evaluates a route, it calls zebra_nhg_rib_find_nhe() to find or create an NHG matching the resolved nexthop state. This goes through zebra_nhe_find() which, when no matching NHG is found by content hash, allocates a new ID and creates a zebra-owned NHG. The route is then updated to reference the new NHG, decrementing the kernel NHG's refcount.

Zebra installs its copy to the dplane:

zebra_nhg_install_kernel() sets the NHG type to ZEBRA_ROUTE_NHG and calls dplane_nexthop_add() to push it to grout.

Zebra deletes the kernel NHG when no longer referenced:

When the kernel NHG's refcount drops to 0, a keep-around timer is started. When it fires, zebra_nhg_uninstall_kernel() sends a delete to the dplane. In grout, this delete is a no-op since the original kernel ID no longer exists (it was already overwritten).