Extend report lldp into a cabling check: compare live LLDP neighbours of nodes and SONiC switches with NetBox #2877

Category: feature | Scope: Large | Related: #2876

Summary

Turn osism report lldp into a cabling check. The command collects the LLDP
neighbours of every physical port live over SSH, on OSISM-managed Linux
nodes and on SONiC switches, and compares what is actually plugged in with the
cabling documented in NetBox. Each port gets a verdict, so a missing cable, a
cable in the wrong port and two swapped cables are visible at a glance.

Devices are selectable with the same NetBox filters and name patterns as
osism report transceiver (#2876). The comparison with NetBox can be switched
off; the command is then a plain LLDP neighbour report, as today, but for
nodes and switches and with CSV/JSON output.

Motivation

Cabling errors are the most common fault after racking and after hardware
replacement, and they are slow to find: a swapped pair of cables still gives
link on both ports, the node often still boots, and the symptom shows up
later as a BGP session to the wrong leaf, a broken bond or asymmetric traffic.
NetBox already holds the intended cabling, and both the nodes and the switches
already speak LLDP. What is missing is the comparison.

Today osism report lldp prints the raw neighbours of the nodes only. Checking
a rack means reading that table next to NetBox, translating EthernetN/alias
names by hand, and logging in to the switches separately.

Analysis of the current code base

osism report lldp today

osism/commands/report.py, class Lldp:

• Hosts come from ansible-inventory --list [--limit], so only Linux nodes.
• One ssh ... lldpctl -f json per host, serially, 30 s timeout, as
  settings.OPERATOR_USER without sudo.
• The parser already handles the lldpctl -f json quirk that one interface is
  a dict and several are a list.
• Output is a tabulate table with Host, Local Interface, Remote Switch,
  Remote Port, Port Description, Age. No other format, no NetBox access.
• Covered by tests/unit/commands/test_report.py (test_lldp_*).

What the switches advertise

The SONiC config generator determines what a node sees as its neighbour:

• files/sonic/config_db.json enables the lldp feature and sets
  intf_naming_mode to standard.
• osism/tasks/conductor/sonic/config_generator.py sets every port's alias
  through convert_sonic_interface_to_alias(), i.e. to the NetBox-style name
  (Eth1/1, Eth1/1/3 for breakout ports), not to the vendor alias from the
  port_config file (twentyFiveGigE1).
• The hostname in DEVICE_METADATA is get_device_hostname(device): the
  custom field inventory_hostname, else the NetBox device name.
• No port description is generated.

SONiC advertises the port alias as LLDP port ID. On a switch configured by
OSISM the remote port a node sees should therefore already be the NetBox
interface name. Switches that were configured differently advertise
EthernetN or the vendor alias; both can be translated with the existing
helpers in osism/tasks/conductor/sonic/interface.py
(convert_netbox_interface_to_sonic(), get_port_config()).

Cabling in NetBox

osism/tasks/conductor/sonic/connections.py already reads cabling the right
way for this purpose: _get_connected_endpoint() and
get_connected_device_via_interface() use interface.connected_endpoints
together with connected_endpoints_reachable. That is the far end of the
complete cable path, through patch panels, which is exactly the device an
LLDP frame comes from. link_peers would return the patch panel.

Those helpers work on one device at a time through the per-thread interface
cache and return only the device. The check needs the peer interface as
well and has to load many devices, so it should fetch the interfaces in bulk
(dcim.interfaces.filter(device_id=[...]), chunked) and read the endpoints
from the result.

Shared groundwork with #2876

Everything around the actual comparison is the same as for
report transceiver: NetBox-driven device selection and classification into
nodes and switches, name patterns, inventory fallback without NetBox, one SSH
session per device run in parallel, SONiC address resolution, mapping of
device-side interface names to NetBox interfaces, and table/CSV/JSON
rendering. This issue assumes the helpers proposed there under
osism/utils/hardware/. Whichever command is implemented first introduces
them; the second one reuses them.

Proposed command

osism report lldp [NAME ...]
    [--no-compare]
    [--type {all,node,switch}] [--include-unmanaged] [--no-mgmt]
    [-l PATTERN | --limit PATTERN]
    [--site SLUG] [--role SLUG] [--rack NAME] [--location NAME]
    [--region SLUG] [--site-group SLUG] [--tenant SLUG] [--tag SLUG]
    [--manufacturer SLUG] [--device-type SLUG] [--platform SLUG]
    [--status STATUS] [--filter KEY=VALUE]
    [--interface PATTERN] [--result RESULT] [--problems]
    [--format {table,csv,json}] [--column NAME] [--long] [--output FILE]
    [--parallel N] [--timeout SECONDS] [--strict]

Examples:

# check everything, show only what is wrong
osism report lldp --problems

# one rack after hardware was replaced
osism report lldp --rack R12 --site fra1

# all compute nodes, fail the calling script on any finding
osism report lldp -l compute --problems --strict

# plain neighbour listing of the spines, no NetBox comparison
osism report lldp --role spine --no-compare --format csv

Example output:

+-------------+-----------+------------------+------------------+--------------+----------------------------+
| Device      | Interface | Expected         | Seen             | Result       | Hint                       |
|-------------+-----------+------------------+------------------+--------------+----------------------------|
| compute-041 | ens1f0    | leaf-1a:Eth1/11  | leaf-1a:Eth1/11  | ok           |                            |
| compute-041 | ens1f1    | leaf-1b:Eth1/11  | leaf-1b:Eth1/11  | ok           |                            |
| compute-042 | ens1f0    | leaf-1a:Eth1/12  | leaf-1b:Eth1/12  | swapped      | swapped with ens1f1        |
| compute-042 | ens1f1    | leaf-1b:Eth1/12  | leaf-1a:Eth1/12  | swapped      | swapped with ens1f0        |
| compute-043 | ens1f0    | leaf-1a:Eth1/13  |                  | missing      | link down                  |
| compute-043 | ens1f1    | leaf-1b:Eth1/13  | leaf-1b:Eth1/14  | wrong-port   | documented at compute-044  |
| compute-044 | ens2f0    |                  | leaf-1a:Eth1/20  | unexpected   | no cable in NetBox         |
+-------------+-----------+------------------+------------------+--------------+----------------------------+

Date: 2026-11-11 12:00:00
Devices: 4   Ports: 7
ok: 2   swapped: 2   missing: 1   wrong-port: 1   unexpected: 1

Device selection and filtering

Identical to #2876, including the decisions taken there:

• NetBox query with status=active plus the typed filter options and
  --filter KEY=VALUE; positional NAME patterns (fnmatch) on device name
  and inventory hostname; --limit as Ansible pattern for nodes; --type.
• A device is a switch when its role is in DEFAULT_SONIC_ROLES and it
  matches NETBOX_FILTER_CONDUCTOR_SONIC; other SONiC-role switches are
  skipped with a warning unless --include-unmanaged is given. A device is a
  node when its hostname is in the Ansible inventory.
• Only the selected devices are logged in to. The peer of a selected port is
  not queried just because it is the peer; it only contributes its own view
  when it is part of the selection as well.
• -l/--limit keeps working as it does today.

Collection

One SSH session per device, devices in parallel (--parallel, default 10,
--timeout per device, default 60 s).

Per neighbour the normalised record holds system name, chassis ID, port ID
with its subtype, port description, management addresses and age. A port can
have more than one neighbour (hypervisor bridges, NIC firmware agents); all
are kept and the comparison treats the port as matching when one of them is
the expected peer.

The link state is collected because it changes the meaning of "no neighbour":
link down means no cable or a dead peer, link up means the peer is silent.

Expected cabling from NetBox

For every selected device the interfaces are loaded in bulk. Virtual and LAG
interfaces are ignored (LLDP runs on the member ports, and NetBox cables are
attached to them).

Management interfaces (mgmt_only) are part of the default run, so the
out-of-band cabling of the switches is checked as well. The rule is the same
for both device kinds: a mgmt_only interface is checked when it has a live
counterpart on the device. On a switch that is the management port eth0,
matched by MAC or name, or as the only mgmt_only interface of the device. A
node's BMC port has no counterpart in the operating system and is skipped
silently instead of being reported as missing; its cable is still verified
from the other side when the OOB switch it is plugged into is a selected SONiC
switch, because there it is an ordinary front-panel port. --no-mgmt leaves
all mgmt_only interfaces out.

The expectation for an interface is its connected_endpoints:

• reachable endpoint of type dcim.interface: expected peer device and
  interface. Several endpoints (breakout cable modelled on one interface) are
  all accepted.
• cable set but the path does not end at a device interface (ends at a
  patch panel, circuit or provider network): no expectation, flagged as an
  incomplete path in the hint.
• no cable: nothing expected.

Resolving LLDP identities to NetBox

Neighbour device: the system name is compared case-insensitively, as given
and with the domain stripped, against NetBox device names and the
inventory_hostname custom field. Fallbacks are the management address
against primary/OOB IPs and the chassis ID against interface MACs. A
neighbour that resolves to nothing is reported as it announced itself.

Neighbour port, on the resolved device, first match wins:

1. port ID or port description equals a NetBox interface name or label,
2. the remote device is a SONiC switch and the port ID is EthernetN or a
   vendor alias: translate through the HWSKU port_config and the index
   built with convert_netbox_interface_to_sonic(),
3. port ID is a MAC address (the lldpd default on Linux): equals a MAC of a
   NetBox interface,
4. the remote device was queried as well: match against its live interface
   list (name, altnames, permanent MAC, alias) and take the NetBox interface
   that live interface is mapped to.

Local port: the mapping defined in #2876 (port-config/alias/name for
switches, mac/name/label for nodes).

Comparison

One result per physical port of a selected device:

Ports with no cable in NetBox, no neighbour and no link are not listed unless
--long is given.

Swap detection works on the whole selection, not per device. For a mismatching
interface I that sees endpoint N, look up the interface J for which
NetBox documents N. If J in turn sees the endpoint documented for I, the
two are swapped. Longer rotations are found by following that chain. This
covers two ports crossed on one node as well as the cables of two neighbouring
nodes crossed on the switch. When J was not queried the finding stays
wrong-port/wrong-device and the hint says where the seen cable is
documented.

The comparison is a pure function over plain data (expected endpoints,
observed neighbours, link states), independent of SSH and pynetbox, so the
whole result matrix is unit-testable.

Without the NetBox comparison

--no-compare skips the comparison; it is also the automatic behaviour when
NetBox is not configured (then only nodes from the inventory can be queried,
as in #2876). The output is the neighbour listing with the columns of today's
command. NetBox filters still work with --no-compare when NetBox is
available, because selection and comparison are independent.

Two checks need no source of truth and stay active: one-sided when both ends
were queried, and no-lldp for ports with link but without neighbour.

Output

• --format table (default), csv, json; --column, --long,
  --output FILE; rows sorted by device and natural interface order.
• Table footer: date, devices, ports, and the count per result.
• --problems hides ok; --result RESULT (repeatable) selects specific
  results; --interface PATTERN filters by device-side or NetBox name.
• JSON is nested per device:

  {
    "generated_at": "2026-11-11T12:00:00Z",
    "compared_with_netbox": true,
    "summary": {"ok": 2, "swapped": 2, "missing": 1},
    "devices": [
      {"name": "compute-042", "kind": "node",
       "interfaces": [
         {"name": "ens1f0", "netbox_name": "ens1f0", "mapping": "mac",
          "oper_status": "up",
          "expected": {"device": "leaf-1a", "interface": "Eth1/12"},
          "seen": [{"device": "leaf-1b", "interface": "Eth1/12",
                    "system_name": "leaf-1b", "port_id": "Eth1/12",
                    "chassis_id": "...", "age": "3 days"}],
          "result": "swapped", "hint": "swapped with ens1f1"}]}
    ],
    "failed": [{"name": "...", "error": "timeout"}]
  }

Error handling and exit code

• An unreachable device goes to the failed list; the report continues. Its
  ports can still appear from the other side when the peers were queried.
• lldpd not running or lldpctl missing is a per-device failure with a
  clear message, not an empty neighbour list.
• Exit code as decided for Add report transceiver command: live NIC and transceiver hardware inventory for Linux nodes and SONiC switches #2876: 1 when selection fails or nothing
  matches, otherwise 0. --strict returns 1 when a device failed or any
  port has a result other than ok.
• The command is read-only on the devices and in NetBox.

Implementation outline

Suggested order, each step a reviewable pull request:

1. Capture real LLDP data (see below), neighbour normalisation for nodes and
   switches, --no-compare listing with selection, parallel collection and
   output formats.
2. Expected cabling from NetBox, identity resolution, results ok,
   missing, no-lldp, wrong-port, wrong-device, unexpected,
   unmapped.
3. Swap detection, one-sided, stale, hints, --problems, --strict.

To confirm on real hardware first (not verified against a live device for
this issue), and to commit as fixtures:

• which port ID and subtype a SONiC switch configured by OSISM advertises
  (expected: the alias, Eth1/1), including breakout ports,
• which port ID and port description lldpd advertises on the nodes as
  deployed by OSISM (MAC or interface name), in particular for bond members,
• whether the system name arrives short or fully qualified,
• the field names of LLDP_ENTRY_TABLE on the SONiC release in use, and
  whether the neighbour of the management port eth0 is listed there.

Acceptance criteria

• osism report lldp collects LLDP neighbours live from managed nodes
  and SONiC switches, in parallel.
• With NetBox available, every port of a selected device that has a cable
  in NetBox or a live neighbour gets one of the results above.
• A removed cable is reported as missing, a cable in the wrong port as
  wrong-port, a cable to another device as wrong-device, an
  undocumented cable as unexpected.
• Two exchanged cables are reported as swapped on both ports with the
  partner named, for a swap within one device and across two devices.
• Cable paths through patch panels are compared with the far-end device.
• The management port of a switch is checked in the default run; BMC
  ports of nodes produce no finding on the node; --no-mgmt excludes
  all mgmt_only interfaces.
• Only selected devices are logged in to; documented peers outside the
  selection are not queried.
• --no-compare produces the neighbour listing without NetBox cabling
  data; without NetBox configured this is the automatic behaviour.
• The selection options of Add report transceiver command: live NIC and transceiver hardware inventory for Linux nodes and SONiC switches #2876 (NAME, --limit, --type, NetBox
  filters, --filter, --include-unmanaged) work identically.
• --problems, --result and --interface filter the rows;
  --format table|csv|json, --column, --long, --output work and
  CSV/JSON on stdout contain no log lines.
• --strict exits 1 on any finding or failed device.
• One unreachable device neither blocks nor aborts the report.
• The comparison and swap detection are covered by unit tests over a
  result matrix; the parsers by fixtures captured from real devices.

Decisions

1. Command: the existing osism report lldp is extended; no separate
   command is added.
2. Default: the comparison with NetBox is on whenever NetBox is
   configured; --no-compare turns it off. The default table of the command
   changes accordingly.
3. Peers outside the selection are not logged in to, and there is no
   option to expand the selection to them. one-sided and swap detection
   work on what was selected.
4. Management interfaces: the OOB cabling of the switches is part of the
   default run; --no-mgmt excludes it.
5. Silent peers: devices that never send LLDP end up as no-lldp. That
   is sufficient; no mechanism to exclude such interfaces is added.

Out of scope

• Changing anything in NetBox (creating or correcting cables from the
  observed state).
• Rule-based checks that are not documented in NetBox, such as "both members
  of a bond must end on different switches". Useful as a follow-up on top of
  the collected data.
• Devices that are neither managed nodes nor SONiC switches, and collecting
  LLDP through the BMC.
• Changing the LLDP configuration of nodes or switches.