Skip to main content
Infrastructure

Both Ends Have to Agree: Automating BGP Between FRR and NSX

ANSIBLE VSPHERE

A nested VMware Cloud Foundation lab has two worlds that need to exchange routes: the NSX overlay, where workloads sit behind a Tier-0 gateway, and everything else — the management network, the gateway, the path out. The join between them is a single eBGP session: the gateway’s FRR talking to the NSX Tier-0. One session. And it took me longer to get right than things with far more moving parts, because a BGP session is the one piece of the lab that only works if two completely different systems agree on exactly the same set of numbers.

Lab Environment

The gateway — the same Linux host that runs the lab’s certificate authority and Keycloak — is also its router. FRR (Free Range Routing) gives it a BGP control plane on top of the inter-VLAN routing it already does. On the other side, NSX stands up an Edge cluster and a Tier-0 gateway with an uplink onto a small peering VLAN. The two meet over one eBGP session, and north-south traffic for the overlay rides across it.

A Session Is an Agreement

eBGP is symmetric in a way that punishes mistakes. Each end declares its own autonomous-system number and the AS number it expects from its neighbour — and those have to be mirror images: my local-AS here must equal your neighbour-AS there, and the other way round. The peer addresses have to match interfaces that actually exist. Transpose one digit in an AS number and the session never leaves Idle. Get the export policy wrong and it comes up, looks healthy, and carries nothing. There is no partial credit, and — this is the part that costs you evenings — most of the failure modes are silent.

Two Worlds, Configured Two Ways

What makes this awkward to automate is that the two ends could not be more different to configure. The gateway end is a text file: Ansible renders bgpd.conf from a Jinja template and drops it into /etc/frr. The NSX end is an API: the automation drives the NSX policy model — an Edge cluster, a Tier-0 gateway, a locale service, a BGP neighbour — through the SDK, object by object. A templated config file on one side and a declarative policy API on the other, and somehow they have to end up describing the same session, the same way, every time the lab is rebuilt.

One Table, Both Ends

The way they stay in agreement is the idea the rest of this lab runs on: the design document is the source of truth. The physical design carries one small table — the BGP peering — with a row per end. The gateway’s FRR config reads it; the NSX Tier-0 role reads it. The Tier-0 takes its own row and points at the gateway’s; the gateway takes its row and points at the Tier-0’s. Both ends are just projections of the same two rows, so they can’t drift: change an AS number or a peer address in the table and both sides move together on the next run. Neither config restates a number — they look it up.

An honest aside, because this post is what made me check: the NSX end wasn’t actually doing that. The gateway’s FRR config had always read the table, but the NSX side had the same numbers typed into its role defaults — correct, and matching, but by hand. They agreed only because I’d kept them in sync. So the first thing writing this produced was a fix: wire the Tier-0’s AS numbers and peer address to the very same table the FRR side reads. Now the agreement is structural rather than something I maintain — which is the whole point. A BGP session is the last place you want two sources of truth.

The Defaults That Bite

Getting the session to actually pass traffic meant unlearning two FRR defaults, both of which fail silently — the worst kind:

  • FRR ships in integrated-config mode, where every daemon reads one combined frr.conf and ignores its own bgpd.conf. Template bgpd.conf in that mode and it has precisely zero effect: bgpd starts with no configuration and the playbook still reports success. The fix is to switch FRR to per-daemon mode so bgpd.conf is read at all. Nothing tells you it didn’t take until you inspect the running config.
  • Modern FRR defaults ebgp-requires-policy to on — RFC 8212, sensible for a transit ISP, wrong for a single-peer lab. The session establishes, looks perfectly healthy, and sends the neighbour nothing; the advertised-prefix count shows the literal word (Policy) where a number should be. One line turns it off and the routes flow.

Each cost an evening, and each is now codified in the role with a comment explaining why — because “why is this one line here?” is exactly the kind of thing a well-meaning cleanup deletes. The logical design records both as decisions with their rationale, so the reasoning survives even if the code is refactored.

Coming Up at Different Times

The two ends don’t just live in different systems — they’re born at different times. The gateway’s FRR exists from almost the first phase of the build; the NSX Tier-0 doesn’t exist until the fabric is stood up much later. So the session genuinely cannot establish until both halves are present and the Tier-0 is realised, and the automation has to treat “not yet” as normal rather than as failure. It also runs active/standby across two Edge VMs, so only one owns the session at a time and BFD hands it over on failure.

When the build does check the session, it verifies rather than assumes — established, and a non-zero advertised count. Even there NSX left a trap: the API that lists advertised routes returns an empty object while it is advertising, so the check reads the sent-prefix counter off the BGP summary instead. One more silent-by-default behaviour, codified once so it can’t mislead the next run.

The Lesson

BGP is the cleanest illustration I have of why design-driven automation is worth the effort. A session is, by definition, an agreement between two parties; if each party’s configuration comes from a different place, they drift, and the failure is quiet — Idle, or established-but-empty, with nothing pointing at the cause. Driving both ends from one row in one table doesn’t make the routing cleverer. It makes the agreement structural — the two sides are the same fact, written once, projected into a Jinja template on one end and a policy API on the other. The hard part was never BGP. It was getting two very different systems to keep telling the same story, every rebuild, without me standing in the middle relaying it.

The lab — the physical design’s BGP table, the FRR templates, and the NSX Tier-0 automation — is at github.com/darrylcauldwell/dda-vcf.