From 1a6dfe0709ab1bfcd5dd39a0ff4ca1726df4c5ab Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Mon, 5 Oct 2026 05:36:53 +0000 Subject: [PATCH 1/3] docs(runbook): name the switch's firewall rules, and delete a stale one in the window (#444) Phase 2 named the pass for the switch UI only as "the rule in network.md". On morpheus it is Allow HTTP to LAN Switch on the Hicks interface. Step 1 now names it and says to copy it with port 443 rather than edit it. Step 7 deletes the HTTP original. pfctl -sr also lists Allow blackbox probe from Prometheus to Switch on the Winterfell interface, 10.0.99.20 to 10.7.7.2:80. blackbox.yaml says that rule was dropped and no probe uses it. A new step 8 deletes it once plain www is off, and leaves the SNMP rule beside it. Refs #444 Co-Authored-By: Claude Opus 5.5 --- docs/runbooks/swap-the-switch.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/runbooks/swap-the-switch.md b/docs/runbooks/swap-the-switch.md index 993b905a..c535545a 100644 --- a/docs/runbooks/swap-the-switch.md +++ b/docs/runbooks/swap-the-switch.md @@ -224,7 +224,10 @@ House offline. Alertmanager silenced. The MokerLink stays on the bench, cabled and powered, until Phase 3 passes. 1. **Widen the firewall pass to `443`** — the rule in `network.md` that admits - `10.7.7.2:80`. Both ports open for the duration of the window. + `10.7.7.2:80`. Both ports open for the duration of the window. On + `morpheus` it is *Allow HTTP to LAN Switch* on the Hicks interface, read + from `pfctl -sr` on 2026-10-05. Copy it beside itself with the destination + port set to HTTPS, rather than editing it. 2. Rack the CRS326 at U9. Cat6 from `morpheus`'s `igc0` to **port 1**, the trunk. 3. Move the patch leads, following the §1.1 map. @@ -235,7 +238,14 @@ and powered, until Phase 3 passes. lab on VLAN 30. 6. Confirm the SNMP scrape is up and the `switch-ui` probe is green. 7. **Only now**, disable plain `www` on the switch and narrow the firewall rule - from `80` to `443`. Prove the UI again afterwards. + from `80` to `443`: delete the HTTP original and keep the HTTPS copy. Prove + the UI again afterwards. +8. **Delete *Allow blackbox probe from Prometheus to Switch*** on the Winterfell + interface. It passes `10.0.99.20 → 10.7.7.2:80`. The comment in + [`blackbox.yaml`](../../stacks/observability/prometheus/targets/blackbox.yaml) + says that rule was dropped and no probe uses it, but `pfctl -sr` on + `morpheus` still listed it on 2026-10-05. Once `www` is off it reaches + nothing. Leave *Allow SNMP from Prometheus to Switch* beside it. If any of 4–6 fails and is not fixed within the window's budget, roll back: the MokerLink returns to U9, the patch leads go back by the same map, and the From 8d4407574ec8519de63768e7b76f96f36c97882d Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Mon, 5 Oct 2026 05:39:02 +0000 Subject: [PATCH 2/3] docs(runbook): Phase 2 checks the SNMP scrape, not a switch-ui probe that was removed (#444) Step 6 said to confirm the switch-ui probe was green. That probe was removed on 2026-09-06, as blackbox.yaml records, so the step could not be passed as written. It now checks only up{job="snmp"}, because step 4 already proves the UI. The Phase 3 item about the probe is left to #653, which rewrites it. Refs #444 Co-Authored-By: Claude Opus 5.5 --- docs/runbooks/swap-the-switch.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/runbooks/swap-the-switch.md b/docs/runbooks/swap-the-switch.md index c535545a..f2a0f184 100644 --- a/docs/runbooks/swap-the-switch.md +++ b/docs/runbooks/swap-the-switch.md @@ -236,7 +236,11 @@ and powered, until Phase 3 passes. `morpheus`, which depends on the switch you have just replaced. 5. Walk the VLANs: internet, wireless, a camera on Skids, a host on VLAN 99, the lab on VLAN 30. -6. Confirm the SNMP scrape is up and the `switch-ui` probe is green. +6. Confirm the SNMP scrape is up: `up{job="snmp"}` for `10.7.7.2` is `1`, over + v3 once the repository half is applied. There is no `switch-ui` probe to + check. It was removed on 2026-09-06, as the comment in + [`blackbox.yaml`](../../stacks/observability/prometheus/targets/blackbox.yaml) + records, and the UI is proven by step 4. 7. **Only now**, disable plain `www` on the switch and narrow the firewall rule from `80` to `443`: delete the HTTP original and keep the HTTPS copy. Prove the UI again afterwards. From 8b9b74852d9b8c448bc4da2ae20cbc3722a81f49 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Mon, 5 Oct 2026 05:40:08 +0000 Subject: [PATCH 3/3] docs(runbook): Phase 3 adds no switch-ui probe, because none exists to move (#444) Phase 3 said to move the switch-ui blackbox target from http to https. Both probes were removed on 2026-09-06, so there is nothing to move. This takes #653's wording for the item, decided 2026-09-23: the probe is not brought back, and the leaf's expiry becomes a dated row in successor-handover.md instead. The last sentence says this phase adds that row, because on main it does not exist yet. Refs #444 Co-Authored-By: Claude Opus 5.5 --- docs/runbooks/swap-the-switch.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/runbooks/swap-the-switch.md b/docs/runbooks/swap-the-switch.md index f2a0f184..5d9f6136 100644 --- a/docs/runbooks/swap-the-switch.md +++ b/docs/runbooks/swap-the-switch.md @@ -277,12 +277,15 @@ all of it is the point. `network.rules.yaml`. **Do not carry the port references across unchecked**: this is a 24 + 2 device replacing a 26-port one, so `ifIndex` and `ifName` change, and any rule or dashboard panel naming a port needs re-deriving. -- The `switch-ui` blackbox target and its `via: dns` twin, `http` → `https`, - with a `ca_file` rather than `insecure_skip_verify` — the estate CA is already - how blackbox verifies Grafana. - [`blackbox.test.yaml`](../../stacks/observability/prometheus/tests/blackbox.test.yaml) - uses `switch-ui` as its worked example of an endpoint with no dns twin; that - needs a different subject. +- **No `switch-ui` probe, and none is added.** An earlier draft of this list + said to move it from `http` to `https`. There was nothing to move: both + probes were removed on 2026-09-06, when the switch LAN was closed to VLAN 99 + apart from SNMP. `prometheus/targets/blackbox.yaml` records why, and bringing + one back would need a new `10.0.99.20 → 10.7.7.2:443` pass, which is a + segmentation decision. Decided 2026-09-23 to leave it out: the SNMP scrape + already watches the switch. The cost is that the leaf's expiry pages nobody, + so this phase adds a dated row for it to + [`successor-handover.md`](successor-handover.md#what-fails-soonest-if-nobody-touches-anything). - `SNMP_COMMUNITY_MOKERLINK` in `secrets/observability.sops.yaml` has no consumer once v2c is off, and its name is a misnomer the moment `neo` is a MikroTik. Retiring it touches `observability.example.yaml`,