Skip to content

docs(runbook): name the switch's firewall rules, and delete a stale one in the window (#444) #913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Gerrrt merged 3 commits into from
Oct 5, 2026
Merged

docs(runbook): name the switch's firewall rules, and delete a stale one in the window (#444) #913

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1a6dfe0
docs(runbook): name the switch's firewall rules, and delete a stale o…
Gerrrt Oct 5, 2026
8d44075
docs(runbook): Phase 2 checks the SNMP scrape, not a switch-ui probe …
Gerrrt Oct 5, 2026
8b9b748
docs(runbook): Phase 3 adds no switch-ui probe, because none exists t…
Gerrrt Oct 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 26 additions & 9 deletions docs/runbooks/swap-the-switch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,10 @@ House offline. Alertmanager silenced. The MokerLink stays on the bench, cabled
and powered, until Phase 3 passes.

1. **Widen the firewall pass to `443`** — the rule in `network.md` that admits
`10.7.7.2:80`. Both ports open for the duration of the window.
`10.7.7.2:80`. Both ports open for the duration of the window. On
`morpheus` it is *Allow HTTP to LAN Switch* on the Hicks interface, read
from `pfctl -sr` on 2026-10-05. Copy it beside itself with the destination
port set to HTTPS, rather than editing it.
2. Rack the CRS326 at U9. Cat6 from `morpheus`'s `igc0` to **port 1**, the
trunk.
3. Move the patch leads, following the §1.1 map.
Expand All @@ -233,9 +236,20 @@ and powered, until Phase 3 passes.
`morpheus`, which depends on the switch you have just replaced.
5. Walk the VLANs: internet, wireless, a camera on Skids, a host on VLAN 99, the
lab on VLAN 30.
6. Confirm the SNMP scrape is up and the `switch-ui` probe is green.
6. Confirm the SNMP scrape is up: `up{job="snmp"}` for `10.7.7.2` is `1`, over
v3 once the repository half is applied. There is no `switch-ui` probe to
check. It was removed on 2026-09-06, as the comment in
[`blackbox.yaml`](../../stacks/observability/prometheus/targets/blackbox.yaml)
records, and the UI is proven by step 4.
7. **Only now**, disable plain `www` on the switch and narrow the firewall rule
from `80` to `443`. Prove the UI again afterwards.
from `80` to `443`: delete the HTTP original and keep the HTTPS copy. Prove
the UI again afterwards.
8. **Delete *Allow blackbox probe from Prometheus to Switch*** on the Winterfell
interface. It passes `10.0.99.20 → 10.7.7.2:80`. The comment in
[`blackbox.yaml`](../../stacks/observability/prometheus/targets/blackbox.yaml)
says that rule was dropped and no probe uses it, but `pfctl -sr` on
`morpheus` still listed it on 2026-10-05. Once `www` is off it reaches
Comment thread
Gerrrt marked this conversation as resolved.
nothing. Leave *Allow SNMP from Prometheus to Switch* beside it.

If any of 4–6 fails and is not fixed within the window's budget, roll back: the
MokerLink returns to U9, the patch leads go back by the same map, and the
Expand Down Expand Up @@ -263,12 +277,15 @@ all of it is the point.
`network.rules.yaml`. **Do not carry the port references across unchecked**:
this is a 24 + 2 device replacing a 26-port one, so `ifIndex` and `ifName`
change, and any rule or dashboard panel naming a port needs re-deriving.
- The `switch-ui` blackbox target and its `via: dns` twin, `http` → `https`,
with a `ca_file` rather than `insecure_skip_verify` — the estate CA is already
how blackbox verifies Grafana.
[`blackbox.test.yaml`](../../stacks/observability/prometheus/tests/blackbox.test.yaml)
uses `switch-ui` as its worked example of an endpoint with no dns twin; that
needs a different subject.
- **No `switch-ui` probe, and none is added.** An earlier draft of this list
said to move it from `http` to `https`. There was nothing to move: both
probes were removed on 2026-09-06, when the switch LAN was closed to VLAN 99
apart from SNMP. `prometheus/targets/blackbox.yaml` records why, and bringing
one back would need a new `10.0.99.20 → 10.7.7.2:443` pass, which is a
segmentation decision. Decided 2026-09-23 to leave it out: the SNMP scrape
already watches the switch. The cost is that the leaf's expiry pages nobody,
so this phase adds a dated row for it to
[`successor-handover.md`](successor-handover.md#what-fails-soonest-if-nobody-touches-anything).
- `SNMP_COMMUNITY_MOKERLINK` in `secrets/observability.sops.yaml` has no
consumer once v2c is off, and its name is a misnomer the moment `neo` is a
MikroTik. Retiring it touches `observability.example.yaml`,
Expand Down
Loading