Run:ai platform: LoadBalancer address and opt-in Gateway API exposure, with tests

config.example/group_vars/k8s_cluster.yml

@@ -115,3 +115,9 @@ runai_mode: "off"
 # Lab without DNS: let the role add the hosts entry and a cluster DNS rewrite.
 #runai_platform_manage_hosts_entry: true
 #runai_platform_coredns_rewrite: true
+# How the control plane is reached: ingress (HAProxy, default) or gateway (Kubernetes
+# Gateway API with KGateway). With ingress and a LoadBalancer service the role waits for
+# the external address (deploy a load balancer first: scripts/k8s/deploy_loadbalancer.sh).
+#runai_platform_exposure: ingress
+#runai_platform_ingress_service_type: LoadBalancer   # or NodePort
+#runai_gateway_workload_routing: path               # gateway only: path or host

docs/k8s-cluster/ingress.md

@@ -21,6 +21,8 @@ DeepOps provides scripts you can run to configure a simple Load Balancer and/or
 
 Modify `config/helm/metallb.yml` to configure the IP range that the load balancer will hand out.
 
+With IPVS kube-proxy (Kubespray's default), MetalLB L2 requires `kube_proxy_strict_arp: true` before deployment; see [strict ARP setup and existing-cluster steps](runai.md#reaching-the-control-plane-loadbalancer-or-gateway-api).
+
 Run the script to deploy the load balancer:
 
 ```bash

docs/k8s-cluster/runai.md

@@ -26,8 +26,10 @@ and any DeepOps-owned UI. The web UI in `platform` mode is the upstream one.
 - `platform`, which you must provide before running the role:
   - a default StorageClass (`playbooks/k8s-cluster/nfs-client-provisioner.yml`)
   - a load balancer for a `LoadBalancer` service ([Load Balancer and Ingress](ingress.md)),
-    or set `runai_platform_ingress_service_type: NodePort` (only NodePort was
-    live-tested in 26.11; `LoadBalancer` is untested in 26.11)
+    or set `runai_platform_ingress_service_type: NodePort` (26.11 live-tested
+    NodePort, and `LoadBalancer` with MetalLB on a single node for both the
+    ingress and the Gateway API paths; reaching that address from another
+    machine was not part of the test)
   - `kube-prometheus-stack` 88.6.5 or newer (`scripts/k8s/deploy_monitoring.sh`
     with `HELM_PROMETHEUS_CHART_VERSION` set)
   - a DNS name for the control plane, and a TLS certificate for it
@@ -44,6 +46,8 @@ and any DeepOps-owned UI. The web UI in `platform` mode is the upstream one.
 | gpu-fractioning (bundled in the KAI chart) | v0.1.7 |
 | Run:ai control-plane and cluster charts | 3.0.20 |
 | HAProxy ingress controller chart (platform) | 1.54.2 |
+| Gateway API CRDs (`runai_platform_exposure: gateway`) | v1.4.0 |
+| KGateway controller and CRD charts (gateway) | v2.2.1 |
 
 All are Apache-2.0. The Run:ai images come from `nvcr.io` without an NGC key.
 Versions are variables (`runai_*_version`); pins are refreshed with each release.
@@ -163,6 +167,70 @@ The role then:
 4. installs the control plane, registers the cluster with it through its API, and
    installs the cluster chart with the short-lived install key the API returns.
 
+### Reaching the control plane: LoadBalancer or Gateway API
+
+`runai_platform_exposure` picks how traffic gets in. The default, `ingress`, is the
+HAProxy path above and is unchanged.
+
+**LoadBalancer.** With `runai_platform_ingress_service_type: LoadBalancer` (the
+default) the role waits up to `runai_platform_address_timeout` seconds (300) for the
+service to get an external address and uses it, instead of the node address, for the
+hosts entry, the CoreDNS entry and the API helper. Install a load balancer first:
+`./scripts/k8s/deploy_loadbalancer.sh` with an address pool in `config/helm/metallb.yml`
+([Load Balancer and Ingress](ingress.md)). If no address appears the role stops and says
+so; set `runai_platform_ingress_ip` to use a fixed address instead.
+
+**MetalLB L2 with IPVS kube-proxy.** Kubespray defaults to IPVS mode. Before
+deploying the cluster, set the following in `config/group_vars/k8s_cluster.yml`:
+
+```yaml
+kube_proxy_strict_arp: true
+```
+
+On an existing cluster, edit the `kube-proxy` ConfigMap in `kube-system` and set
+`ipvs.strictARP: true` inside its `config.conf`, then restart kube-proxy so every
+instance loads the change. Keep the group variable above set as well so a later
+Kubespray run preserves it. Without strict ARP, nodes can answer ARP for the
+LoadBalancer address even when they are not the MetalLB L2 announcer, sending
+traffic to the wrong node. This applies to both the ingress LoadBalancer and a
+Gateway backed by a LoadBalancer service; see the
+[upstream MetalLB requirement](https://metallb.io/installation/#preparation).
+
+**26.11 validation scope.** The release validation uses kube-proxy in iptables
+mode, not IPVS. IPVS operation relies on the strict ARP setting above; it is an
+upstream MetalLB requirement, not a separately live-tested path for this release.
+
+**Gateway API** (opt in). Set `runai_platform_exposure: gateway` to follow the
+[Run:ai Gateway API guide](https://docs.nvidia.com/run-ai/self-hosted/v3.0/documentation/installation-and-configuration/config/networking/kubernetes-gateway-api).
+The role then, instead of installing HAProxy:
+
+1. installs the Gateway API CRDs (v1.4.0, standard channel) if none are present, and
+   missing KGateway releases (v2.2.1) in `kgateway-system`, recording ownership
+   in per-release `deepops-runai-kgateway*` ConfigMaps. Existing unowned releases
+   are not upgraded or adopted;
+2. creates one `Gateway` (`runai-gateway`) in the Run:ai cluster namespace with a TLS
+   listener for the domain, using the same certificate as the ingress path;
+3. installs the control plane and cluster with Ingress off and Gateway API on, pointing at
+   that Gateway, and uses the Gateway's address for the lab DNS helpers.
+
+KGateway is the controller in the guide, it is Apache-2.0 and conformant. Any conformant
+controller works: set `runai_gateway_install_controller: false` and
+`runai_gateway_class` to its class; you then install it yourself. Workloads are served
+under URL paths of the one domain (`runai_gateway_workload_routing: path`, the default).
+`host` gives each workload a sub-domain: it needs wildcard DNS and a certificate that covers
+`*.<domain>` (the generated lab certificate does).
+
+Uninstall (`runai_state=absent`, without needing to repeat the exposure option)
+removes the Gateway and discovers the ownership records. It removes owned KGateway
+releases and Gateway API CRDs only when no remaining Gateway, route or KGateway
+policy depends on them. Pre-existing releases and CRDs are preserved. The controller
+namespace is left in place because it may contain unrelated resources; an empty
+namespace alone is not an uninstall failure. `--verify-uninstalled` detects remaining
+ownership records and annotated CRDs, including resources retained for other users.
+To switch
+between `ingress` and `gateway`, uninstall first and install again. Prove the path with
+`--exposure gateway` (or `ingress`) in the validator below.
+
 **Names.** `runai_platform_domain` must resolve to the ingress address from the
 nodes and from wherever you browse. Without DNS (a lab), set
 `runai_platform_manage_hosts_entry: true` (hosts-file line on the control-plane
@@ -176,9 +244,19 @@ Check it (the admin credentials go in the environment, never on the command line
 RUNAI_USERNAME=admin@example.com RUNAI_PASSWORD='...' \
   python3 scripts/validation/validate_runai.py --json --mode platform \
   --platform-domain runai.example.com --ca-file /opt/deepops/runai/ca.pem \
-  --create-project deepops-validate
+  --create-project deepops-validate --exposure ingress
 ```
 
+The result includes `exposure_reached` (`ingress`, `gateway` or `unknown`): the path that
+owns the address the validator connected to, linked to an Ingress for the platform
+domain or an HTTPRoute for that domain and its referenced Gateway in the Run:ai
+namespaces. Node addresses require a NodePort ingress service; a LoadBalancer does
+not claim every node address. Ambiguous or unlinked addresses report `unknown`.
+Override `--cp-namespace` and `--cluster-namespace` if using non-default namespaces.
+`--exposure ingress|gateway` makes `ok` false
+if the control plane was reached any other way, so a Gateway install that quietly still
+answers through the old ingress cannot pass. For the Gateway install use `--exposure gateway`.
+
 In platform mode, pods must be submitted in a Run:ai project's namespace and use
 that project's queue. The validator does this itself: `--create-project deepops-validate`
 creates that project through the API (one GPU of quota in the first node pool), runs the
@@ -223,4 +301,5 @@ too (this deletes any queues and projects still defined).
 | Pod stays `Pending` with `schedulerName: kai-scheduler` | No queue label, or the queue does not exist: `kubectl get queues.scheduling.run.ai`. |
 | A half-GPU pod is rejected | It also asks for a whole `nvidia.com/gpu`, or the fraction is not between 0 and 1. |
 | Validator: scheduler deployments missing | The KAI operator is still starting. Wait a minute and rerun. |
+| Role stops: "No external address" | The LoadBalancer or Gateway got no address. Deploy a load balancer (`scripts/k8s/deploy_loadbalancer.sh`) or use `NodePort` / `runai_platform_ingress_ip`. |
 | Platform validator: control plane unreachable | The domain does not resolve to the ingress address, or the CA is not trusted: pass `--ca-file` and `--resolve-ip`. |

roles/runai/defaults/main.yml

@@ -70,8 +70,13 @@ runai_platform_ingress_repo: "https://haproxytech.github.io/helm-charts"
 runai_platform_ingress_chart_version: "1.54.2"
 runai_platform_ingress_release_name: "haproxy-kubernetes-ingress"
 runai_platform_ingress_namespace: "haproxy-controller"
-# NodePort is the only type live-tested in 26.11; LoadBalancer is untested in 26.11.
+# 26.11 live-tested NodePort and LoadBalancer (MetalLB, single node, ingress and Gateway API).
+# With LoadBalancer the role waits for the external address (MetalLB: see
+# scripts/k8s/deploy_loadbalancer.sh) and uses it for the hosts entry, the CoreDNS
+# entry and the validator, instead of the node address.
 runai_platform_ingress_service_type: "LoadBalancer"
+# Seconds to wait for that external address before the role stops with an explanation.
+runai_platform_address_timeout: 300
 # Single-node labs without a load balancer: serve 80/443 directly on the node.
 runai_platform_ingress_host_ports: false
 
@@ -82,9 +87,38 @@ runai_platform_tls_cert_file: ""
 runai_platform_tls_key_file: ""
 runai_platform_ca_file: ""
 
+# How the control plane is reached from outside the cluster.
+#   ingress  HAProxy ingress controller (default, unchanged)
+#   gateway  Kubernetes Gateway API: the role installs the Gateway API CRDs and the
+#            KGateway controller (unless they are already there), creates one Gateway
+#            and points the Run:ai charts at it. Nothing else changes. Pick one and
+#            uninstall in the same mode; to switch, uninstall first.
+runai_platform_exposure: "ingress"
+
+# Gateway exposure only. Versions follow the Run:ai self-hosted v3.0 Gateway API guide.
+# KGateway is used because the guide uses it, it is Apache-2.0 and conformant, and one
+# controller keeps the live test small. Any conformant controller works: set
+# runai_gateway_install_controller: false and runai_gateway_class to its class.
+runai_gateway_api_version: "v1.4.0"
+runai_gateway_api_crds_url: "https://github.com/kubernetes-sigs/gateway-api/releases/download/{{ runai_gateway_api_version }}/standard-install.yaml"
+runai_gateway_install_controller: true
+runai_gateway_class: "kgateway"
+runai_gateway_controller_version: "v2.2.1"
+runai_gateway_controller_chart: "oci://cr.kgateway.dev/kgateway-dev/charts/kgateway"
+runai_gateway_controller_crds_chart: "oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds"
+runai_gateway_controller_namespace: "kgateway-system"
+runai_gateway_name: "runai-gateway"
+# The Gateway lives in the Run:ai cluster namespace (runai_cluster_namespace).
+# path: workloads share the control-plane domain under URL paths (any certificate).
+# host: workloads get sub-domains; needs a wildcard DNS entry and a wildcard certificate
+# (the generated lab certificate already has one).
+runai_gateway_workload_routing: "path"
+
 # Lab convenience, all off by default. Use when the domain has no DNS record.
-# Address the domain should point at (default: this host's primary address).
-runai_platform_ingress_ip: "{{ ansible_default_ipv4.address | default('') }}"
+# Address the domain should point at. Leave empty to use, in this order: the Gateway's
+# address (gateway exposure), the ingress service's external address (LoadBalancer),
+# else this host's primary address. Set it to override all of those.
+runai_platform_ingress_ip: ""
 # Add "<ingress ip> <domain>" to /etc/hosts on the control-plane host.
 runai_platform_manage_hosts_entry: false
 # Make the domain resolve inside the cluster (CoreDNS rewrite to the ingress

roles/runai/tasks/coredns.yml

@@ -11,7 +11,7 @@
 - name: Plan the Corefile change
   ansible.builtin.set_fact:
     runai_corefile_old: "{{ (runai_corefile_raw.stdout | from_json).data.Corefile }}"
-    runai_corefile_block: "    hosts {\n      {{ runai_platform_ingress_ip }} {{ runai_platform_domain }}\n      fallthrough\n    }  # deepops-runai\n"
+    runai_corefile_block: "    hosts {\n      {{ runai_exposure_address | default('') }} {{ runai_platform_domain }}\n      fallthrough\n    }  # deepops-runai\n"
   no_log: true
 
 - name: Compute the new Corefile

roles/runai/tasks/gateway.yml

@@ -0,0 +1,24 @@
+---
+# One Gateway in the Run:ai cluster namespace, shaped like the Run:ai v3.0 guide. The
+# listeners accept routes from every namespace because the control plane's route lives
+# in its own namespace. Both certificate secrets come from the files used for ingress.
+- name: Create the Gateway certificate secrets
+  ansible.builtin.shell: |
+    set -euo pipefail
+    {{ runai_kubectl }} -n {{ runai_cluster_namespace }} create secret tls {{ item }} \
+      --cert {{ runai_tls_cert | quote }} --key {{ runai_tls_key | quote }} --dry-run=client -o yaml \
+      | {{ runai_kubectl }} apply -f -
+  args:
+    executable: /bin/bash
+  loop: "{{ ['runai-cluster-domain-tls-secret'] + (['runai-cluster-domain-star-tls-secret'] if runai_gateway_workload_routing == 'host' else []) }}"
+  register: runai_gw_tls
+  changed_when: "'created' in runai_gw_tls.stdout or 'configured' in runai_gw_tls.stdout"
+  no_log: true
+  become: true
+
+- name: Create the Gateway
+  ansible.builtin.command: "{{ runai_kubectl }} apply --filename -"
+  args:
+    stdin: "{{ runai_gateway_manifest | to_json }}"
+  register: runai_gw_apply
+  changed_when: "'created' in runai_gw_apply.stdout or 'configured' in runai_gw_apply.stdout"

roles/runai/tasks/gateway_controller.yml

@@ -0,0 +1,48 @@
+---
+# Gateway API CRDs + KGateway controller. Installed only if missing, and marked so
+# uninstall removes exactly what this role added (never a pre-existing install).
+- name: Look for existing Gateway API CRDs
+  ansible.builtin.shell: >-
+    set -o pipefail;
+    {{ runai_kubectl }} get crd --output name | grep -E '\.gateway\.networking\.k8s\.io$' || true
+  args:
+    executable: /bin/bash
+  register: runai_gw_crds_before
+  changed_when: false
+  check_mode: false
+
+- name: Install the Gateway API CRDs (standard channel, pinned)
+  ansible.builtin.command: "{{ runai_kubectl }} apply --filename {{ runai_gateway_api_crds_url }}"
+  environment: "{{ proxy_env if proxy_env is defined else {} }}"
+  register: runai_gw_crds_apply
+  changed_when: true
+  when: runai_gw_crds_before.stdout_lines | length == 0
+
+- name: Mark the CRDs this role installed
+  ansible.builtin.shell: >-
+    set -o pipefail;
+    {{ runai_kubectl }} get crd --output name | grep -E '\.gateway\.networking\.k8s\.io$'
+    | xargs -r -I{} {{ runai_kubectl }} annotate --overwrite {} deepops.nvidia.com/runai-installed=true
+  args:
+    executable: /bin/bash
+  changed_when: true
+  when: runai_gw_crds_apply is not skipped
+
+- name: Look for existing KGateway releases
+  ansible.builtin.command: "{{ runai_helm }} list --all --namespace {{ runai_gateway_controller_namespace }} --output json"
+  register: runai_gw_helm_before
+  changed_when: false
+  check_mode: false
+  when: runai_gateway_install_controller | bool
+
+- name: Note existing KGateway releases
+  ansible.builtin.set_fact:
+    runai_gw_releases_before: "{{ runai_gw_helm_before.stdout | from_json | map(attribute='name') | list }}"
+  when: runai_gateway_install_controller | bool
+
+- name: Install missing or role-owned KGateway releases
+  ansible.builtin.include_tasks: gateway_release.yml
+  loop: [kgateway-crds, kgateway]
+  loop_control:
+    loop_var: runai_gw_release
+  when: runai_gateway_install_controller | bool

roles/runai/tasks/gateway_release.yml

@@ -0,0 +1,25 @@
+---
+# A per-release marker survives reruns; never adopt an existing Helm release.
+- name: "{{ runai_gw_release }} | read ownership record"
+  ansible.builtin.command: "{{ runai_kubectl }} --namespace {{ runai_gateway_controller_namespace }} get configmap deepops-runai-{{ runai_gw_release }} --ignore-not-found --output name"
+  register: runai_gw_owner
+  changed_when: false
+  check_mode: false
+
+- name: "{{ runai_gw_release }} | install only missing or owned release"
+  ansible.builtin.include_tasks: helm_release.yml
+  vars:
+    _rel:
+      release: "{{ runai_gw_release }}"
+      namespace: "{{ runai_gateway_controller_namespace }}"
+      chart: "{{ runai_gateway_controller_crds_chart if runai_gw_release == 'kgateway-crds' else runai_gateway_controller_chart }}"
+      version: "{{ runai_gateway_controller_version }}"
+      helm_values: {}
+  when: runai_gw_release not in runai_gw_releases_before or runai_gw_owner.stdout | length > 0
+
+- name: "{{ runai_gw_release }} | record ownership of new release"
+  ansible.builtin.command:
+    argv: ["{{ runai_kubectl }}", apply, --filename, '-']
+    stdin: "{{ {'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'name': 'deepops-runai-' ~ runai_gw_release, 'namespace': runai_gateway_controller_namespace}, 'data': {'release': runai_gw_release}} | to_json }}"
+  changed_when: true
+  when: runai_gw_release not in runai_gw_releases_before