Add External API endpoints for BGP unnumbered interface manager status by taspelund · Pull Request #10705 · oxidecomputer/omicron

taspelund · 2026-07-01T03:39:04Z

This is the omicron side of maghemite#800.

In mgd, we are refactoring the internals of BGP unnumbered which affects the mg-api by no longer requiring the BGP ASN for querying the state... However, omicron never had plumbing to query BGP unnumbered interface state to begin with, which is what this PR remedies.

The new API endpoints are purely status, with no config whatsoever.

The maghemite hash currently points to the feature branch, but will be updated to main once this PR's counterpart lands.

Now that maghemite has migrated to the RFD 619 convention, it exposes most types via the mg-api-types crate. This crate should have its hash updated alongside mg-admin-client and ddm-admin-client. Signed-off-by: Trey Aspelund <trey@oxidecomputer.com>

taspelund · 2026-07-01T03:40:46Z

I have taken this through a4x2 manually and have validated that e2e connectivity works, as do the new endpoints:

trey@korgano 03:21:19 AM | ~/git/omicron ➦ d372d6342 ○
‣ curl -sS "${AUTH[@]}" "http://198.51.100.24/v1/system/networking/bgp/unnumbered/manager" | jq
[
  {
    "switch_slot": "switch0",
    "state": {
      "monitor_running": true,
      "pending_interfaces": [],
      "active_interfaces": [
        "qsfp0",
        "qsfp1"
      ]
    }
  },
  {
    "switch_slot": "switch1",
    "state": {
      "monitor_running": true,
      "pending_interfaces": [],
      "active_interfaces": [
        "qsfp1",
        "qsfp0"
      ]
    }
  }
]

trey@korgano 03:21:22 AM | ~/git/omicron ➦ d372d6342 ○
‣ curl -sS "${AUTH[@]}" "http://198.51.100.24/v1/system/networking/bgp/unnumbered/interfaces" | jq
[
  {
    "switch_slot": "switch0",
    "interface": {
      "interface": "qsfp0",
      "local_address": "fe80::aa40:25ff:fe25:9735",
      "router_lifetime": 1800,
      "discovered_peer": {
        "address": "fe80::aa40:25ff:fe00:9",
        "discovered_at": "2026-07-01T00:28:26Z",
        "last_advertisement": "2026-07-01T03:23:15Z",
        "router_lifetime": 30,
        "reachable_time": 0,
        "retrans_timer": 0,
        "expired": false,
        "time_until_expiry": "9s 61ms"
      },
      "ndp_state": {
        "tx": true,
        "rx": true
      }
    }
  },
  {
    "switch_slot": "switch0",
    "interface": {
      "interface": "qsfp1",
      "local_address": "fe80::aa40:25ff:fe25:9739",
      "router_lifetime": 1800,
      "discovered_peer": {
        "address": "fe80::aa40:25ff:fe00:11",
        "discovered_at": "2026-07-01T00:28:26Z",
        "last_advertisement": "2026-07-01T03:23:15Z",
        "router_lifetime": 30,
        "reachable_time": 0,
        "retrans_timer": 0,
        "expired": false,
        "time_until_expiry": "9s 398ms"
      },
      "ndp_state": {
        "tx": true,
        "rx": true
      }
    }
  },
  {
    "switch_slot": "switch1",
    "interface": {
      "interface": "qsfp1",
      "local_address": "fe80::aa40:25ff:fe4f:cc12",
      "router_lifetime": 1800,
      "discovered_peer": {
        "address": "fe80::aa40:25ff:fe00:12",
        "discovered_at": "2026-07-01T00:24:15Z",
        "last_advertisement": "2026-07-01T03:23:16Z",
        "router_lifetime": 30,
        "reachable_time": 0,
        "retrans_timer": 0,
        "expired": false,
        "time_until_expiry": "9s 550ms"
      },
      "ndp_state": {
        "tx": true,
        "rx": true
      }
    }
  },
  {
    "switch_slot": "switch1",
    "interface": {
      "interface": "qsfp0",
      "local_address": "fe80::aa40:25ff:fe4f:cc0e",
      "router_lifetime": 1800,
      "discovered_peer": {
        "address": "fe80::aa40:25ff:fe00:10",
        "discovered_at": "2026-07-01T00:24:15Z",
        "last_advertisement": "2026-07-01T03:23:16Z",
        "router_lifetime": 30,
        "reachable_time": 0,
        "retrans_timer": 0,
        "expired": false,
        "time_until_expiry": "9s 549ms"
      },
      "ndp_state": {
        "tx": true,
        "rx": true
      }
    }
  }
]

trey@korgano 03:24:15 AM | ~/git/omicron ➦ d372d6342 ○
‣ curl -sS "${AUTH[@]}" "http://198.51.100.24/v1/system/networking/bgp/unnumbered/interfaces/switch0/qsfp0" | jq
{
  "switch_slot": "switch0",
  "interface": {
    "interface": "qsfp0",
    "local_address": "fe80::aa40:25ff:fe25:9735",
    "router_lifetime": 1800,
    "discovered_peer": {
      "address": "fe80::aa40:25ff:fe00:9",
      "discovered_at": "2026-07-01T00:28:26Z",
      "last_advertisement": "2026-07-01T03:27:15Z",
      "router_lifetime": 30,
      "reachable_time": 0,
      "retrans_timer": 0,
      "expired": false,
      "time_until_expiry": "9s 260ms"
    },
    "ndp_state": {
      "tx": true,
      "rx": true
    }
  }
}

Adds external API endpoints to GET the BGP unnumbered status of interfaces and the unnumbered manager. This provides insights into the Router Discovery state (have we learned about a BGP peer via NDP) and the readiness of an interface that's been configured for BGP unnumbered (does the interface exist in the OS? does it have an IPv6 link-local IP? or is it active?). These are all important when diagnosing failures or delays in the establishment of BGP unnumbered peers, since these are all prerequisites for the BGP FSM to complete connections. This is the omicron side of maghemite#800 Signed-off-by: Trey Aspelund <trey@oxidecomputer.com>

Signed-off-by: Trey Aspelund <trey@oxidecomputer.com>

jgallagher · 2026-07-02T15:37:34Z

+impl super::Nexus {
+    pub async fn bgp_unnumbered_manager_status(
+        &self,
+        _optctx: &OpContext,


Slight typo here (and in the other methods):

Suggested change

_optctx: &OpContext,

_opctx: &OpContext,

That said, I'm not sure these methods should take an argument they don't use. I don't work in the auth area much, but my intuition is that auth checks should be performed at the "lowest level", instead of relying on callers performing them - should we be doing an auth check in this method (and therefore using the opctx arg after all)?

I'm not sure what the right approach is here tbh. I followed the approach that the existing bfd methods used as an example since I wasn't sure what to do for authz

jgallagher · 2026-07-02T15:39:14Z

+        let mut result = Vec::new();
+        for switch_slot in [SwitchSlot::Switch0, SwitchSlot::Switch1] {
+            // Log an error if we only have one scrimlet, but keep going.
+            // We still want to return anything we're able to collect.


We're not guaranteed to have any clients here; from the docs on resolve_mgd_clients():

/// # Errors /// /// Fails if we cannot resolve MGD in DNS. /// /// For any MGD instance we resolve via DNS, if the MGD instance does not know /// its own switch slot, the switch slot -> client mapping for that instance /// will be omitted from the returned map. Callers must not assume an `Ok(_)` /// return value contains any client.

As written, that means if we get back an empty map from self.mg_clients(), we'll return an Ok(Vec::new()) result. Is that okay? I wonder if instead of returning a Vec, we should return something like a BTreeMap<SwitchSlot, UnnumberedManagerStateOrError>? Then we could return something logically like

{ "switch0": "no client found: blah blah error", "switch1": "SomeValidStatus" }

jgallagher · 2026-07-02T15:42:38Z

+        let mut result = Vec::new();
+        for switch_slot in [SwitchSlot::Switch0, SwitchSlot::Switch1] {
+            // Log an error if we only have one scrimlet, but keep going.
+            // We still want to return anything we're able to collect.


Same note here about maybe not having clients, and maybe returning a map keyed by switch slot so we can give back partial success?

I initially thought about returning a map keyed on switch slot, but it looked like other endpoints for switch-specific queries (e.g. lldp) all used the approach of returning a vec of wrapper structs that have switch slot as a member.

If we do go with a map, I think we could get rid of the wrapper structs SwitchUnnumberedManagerState and SwitchUnnumberedInterface since all they do for UnnumberedManagerState and UnnumberedInterface is embed the switch slot.

I don't work in this codebase very often, so I'm happy to go whichever way is preferable to the folks who live here

jgallagher · 2026-07-02T15:45:58Z

+
+fn nexus_interface_name(interface: String) -> String {
+    if let Some(interface) = interface
+        .strip_prefix("tfport")


This method makes me nervous and worried we're baking knowledge into nexus that really doesn't belong there. Some questions:

If we expect all interfaces to be named tfport{X}_0, should we fail instead of returning the interface unchanged?

Is this a sign we're overusing Strings and instead should have something like a TfPort(String) type, with methods for formatting itself either with or without the tfport ... _0 trappings?

Are external API callers expected to use a tfport{X}_0 name?

I assume "tf" is from "tofino" - are we setting ourselves up for future pain whenever we have non-tofino switch?

Maybe most of this is a nonissue since this function is private and only used for API version conversions (or maybe it's even addressing some of these questions as we move the API forward)?

If we expect all interfaces to be named tfport{X}_0, should we fail instead of returning the interface unchanged?

I'm not sure on this front. Today mgd takes a String as its query parameter and does a lookup for an interface with that exact name, without any transforms applied -- which is the same logic that's used for BGP unnumbered peers. So from the mgd side, if the String given to it matches a real interface then it'll return state for that same interface. I'm not sure how mgd would take String A and return String B or how we'd want that to appear... I'd think it'd be better to return whatever we got so we can at least still get some operational state, even if the ifname is mangled.

Is this a sign we're overusing Strings and instead should have something like a TfPort(String) type, with methods for formatting itself either with or without the tfport ... _0 trappings?
I assume "tf" is from "tofino" - are we setting ourselves up for future pain whenever we have non-tofino switch?

I would much prefer a new TfPort type that organizes all of this logic, but I don't know how toronto is going to affect things... I imagine it's probably too early to tell since I don't think we have the equivalent of the tfport driver for toronto yet. If we go through the effort to build a type like that, I would almost prefer that it lives someplace like oxnet so it can be reused by mgd, dpd, tfportd, and nexus.

Are external API callers expected to use a tfport{X}_0 name?

No, external API users supply a name like "qsfp0". I added the tfport formatting logic for this endpoint because the other link/port/peer APIs expect the bare name, not the tfport name. No need to expose the internal implementation detail if we can help it.

Maybe most of this is a nonissue since this function is private and only used for API version conversions (or maybe it's even addressing some of these questions as we move the API forward)?

I think today it's a non-issue, but I would definitely like to have more structure in place around these conversions. I just think it's going to require more coordination across the network component repos.

jgallagher · 2026-07-02T15:49:23Z

+    /// Whether the peer entry has expired
+    pub expired: bool,
+    /// Time until expiry (human-readable), or None if already expired
+    pub time_until_expiry: Option<String>,


Are either of these combinations valid?

expired: true, time_until_expiry: Some(...),

expired: false, time_until_expiry: None,

jgallagher · 2026-07-02T15:49:48Z

+    /// When the peer was first discovered (ISO 8601 timestamp)
+    pub discovered_at: String,
+    /// When the most recent Router Advertisement was received (ISO 8601
+    /// timestamp)
+    pub last_advertisement: String,


Could these be DateTime<Utc> instead of String?

If they're coming from a remote source and we're worried about parsing, it's maybe worth noting that in the doc comment. "Expected to be ISO 8601 timestamps, but presented as-is from the remote router" or something like that.

taspelund requested review from internet-diglett, jgallagher and nicolaskagami July 1, 2026 03:39

taspelund self-assigned this Jul 1, 2026

taspelund added enhancement New feature or request. api Related to the API. networking Related to the networking. rust Pull requests that update rust code labels Jul 1, 2026

taspelund mentioned this pull request Jul 1, 2026

bgp unnumbered: consolidate NDP-backed unnumbered state oxidecomputer/maghemite#800

Open

taspelund added 2 commits June 30, 2026 22:53

Add new endpoints to authz tests

654fe27

Signed-off-by: Trey Aspelund <trey@oxidecomputer.com>

taspelund force-pushed the trey/bgp-unnumbered-status branch from 4736713 to 654fe27 Compare July 1, 2026 16:18

jgallagher reviewed Jul 2, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add External API endpoints for BGP unnumbered interface manager status#10705

Add External API endpoints for BGP unnumbered interface manager status#10705
taspelund wants to merge 3 commits into
mainfrom
trey/bgp-unnumbered-status

taspelund commented Jul 1, 2026

Uh oh!

taspelund commented Jul 1, 2026

Uh oh!

Uh oh!

jgallagher Jul 2, 2026

Uh oh!

taspelund Jul 2, 2026

Uh oh!

jgallagher Jul 2, 2026

Uh oh!

jgallagher Jul 2, 2026

Uh oh!

taspelund Jul 2, 2026

Uh oh!

jgallagher Jul 2, 2026

Uh oh!

taspelund Jul 2, 2026

Uh oh!

jgallagher Jul 2, 2026

Uh oh!

jgallagher Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

taspelund commented Jul 1, 2026

Uh oh!

taspelund commented Jul 1, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants