Skip to content

✨ Update Stacks and new tooling - #273

Open
jschoone wants to merge 16 commits into
mainfrom
feat/one-big-beautiful-stack-change
Open

jschoone wants to merge 16 commits into
mainfrom
feat/one-big-beautiful-stack-change

Conversation

@jschoone

Copy link
Copy Markdown
Contributor

Hi @Nils98Ar and @garloff. I decided to put #270, #259, #255, #252 and #248 together again in this PR since we lost the overview and other PRs were already that old again that they needed to be reworked completely. This PR will solve all of them together.
Mostly generated code but tested, including HCP.
This PR will deprecate the scs2 since the number is no longer needed when using subdirs and continues with scs again.

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

  • squash commits
  • include documentation
  • add unit tests

jschoone and others added 8 commits October 23, 2026 22:58
…cripts

Replace the Makefile and the scripts around it with a justfile and a small set of
scripts in hack/ that work on the per-minor-version layout
(providers/<provider>/<stack>/1-XX/).

- justfile: build, publish, dev, install-cso, matrix, update, add-version,
  generate-resources, generate-clusterstack, generate-cluster,
  generate-image-manifests, generate-docs, migrate-cluster, clean and the tools
  container. flake.nix and Containerfile provide just, helm, kubectl, yq and oras.
- build.sh: builds the release of a Kubernetes minor version (cluster class and
  cluster addon charts, clusteraddon.yaml, metadata). Development builds get the
  version v0-ttl.<timestamp>; the release directory and the OCI tag use the dash
  notation (v0-ttl-<timestamp>), which is what the Cluster Stack Operator
  requests. Stable releases are numbered highest published version + 1, a build
  without changes is not published again, --release-version selects a version.
  --validate checks the addon bundle, --install-cso installs the operator with
  the matching OCI configuration (CSO_CHART, CSO_IMAGE_REPOSITORY/TAG).
- update.sh: Kubernetes patch versions (dl.k8s.io), node images in
  image-manager.yaml (--check-image-repo) and addon chart versions. Minor
  versions with a FROZEN file are skipped.
- add-version.sh: scaffold a new Kubernetes minor version from the latest one.
- generate-resources.sh: ClusterStack (channel stable, or custom for development
  builds) and Cluster. The Cluster uses the API version of the ClusterClass of
  the stack (classRef for v1beta2, class for v1beta1).
- generate-image-manifests.sh (ORC images), show-matrix.sh (version and addon
  matrix).
- docugen.py: configuration documentation from the ClusterClass variables. Only
  the sections between BEGIN/END GENERATED markers are replaced (--update), hand
  written text is kept.
- update-gateway-api-crds.sh: vendors the Gateway API CRDs for the gateway-api
  addon of the scs stacks.

Removes the scripts that are replaced (ensure-*, generate_version.py,
generate_openstack_image_manager_yaml.sh, kind-dev.sh, the release notes tool).

Co-authored-by: Nils Arnold <arnold@aov.de>
Signed-off-by: Jan Schoone <6106846+jschoone@users.noreply.github.com>
Co-authored-by: Nils Arnold <arnold@aov.de>
Signed-off-by: Jan Schoone <6106846+jschoone@users.noreply.github.com>
The scs stack for Kubernetes 1.33 to 1.36 (1.33.11, 1.34.12, 1.35.9, 1.36.5),
every minor version in its own self-contained directory. The ClusterClass uses
cluster.x-k8s.io/v1beta2 and needs Cluster API >= 1.11 and a Cluster Stack
Operator that supports it. Kubernetes 1.36 additionally needs Cluster API
>= 1.12.8 or >= 1.13.1.

- Unified variables: flavor, rootDisk, serverGroupID, ... apply to all nodes and
  are overridden per control plane or machine deployment via
  variables.overrides, instead of separate controlPlane*/worker* variables.
  Node images are managed in image-manager.yaml.
- Cilium with native routing and kube-proxy replacement (1.35, 1.36), Cilium
  1.19.4 (1.33 to 1.35) and 1.20.2 (1.36).
- serviceLoadBalancer (#241): Octavia OVN (default) or Amphora for Services of
  type LoadBalancer. With OVN the cloud controller manager manages the security
  groups, otherwise clients outside of the cluster do not reach the node ports.
- diskPinAZ: create root and additional volumes in the AZ of the machine.
- gateway-api addon: the Gateway API CRDs (v1.4.1 for Cilium 1.19, v1.6.3 for
  Cilium 1.20) are applied before Cilium and the cilium GatewayClass is created,
  Cilium installs neither.
- 1.35 and 1.36: machine health checks and rollout strategy, security groups
  with allowAllInClusterTraffic, containerd limits and registry mirror CA, API
  server audit logging, componentOverrides for the resources of components and
  addons, ciliumHostLegacyRouting, OCCM on the control plane with hostNetwork
  (1.36).

Co-authored-by: Nils Arnold <arnold@aov.de>
Signed-off-by: Jan Schoone <6106846+jschoone@users.noreply.github.com>
scs 1.32 is the only scs version that has been released before. It is kept as it
was (v1beta1 ClusterClass, variables like cloud_name and ssh_key), moved into
1-32/, so that existing clusters can be upgraded within the minor version. The
unified variables and the new features start with 1.33.

The directory is marked with a FROZEN file, hack/update.sh skips it.

Signed-off-by: Jan Schoone <6106846+jschoone@users.noreply.github.com>
… the stack

scs2 is only maintained from now on and will be phased out, scs is the stack to
use.

- 1-33 to 1-36, every minor version in its own directory. The ClusterClass
  templates and the variables (controlPlane*/worker*) are unchanged and still
  v1beta1, only the chart metadata differs, so existing clusters keep working.
- Node images are managed in image-manager.yaml, the per-version versions.yaml,
  kubernetes.yaml and image.yaml are gone.
- DEPRECATED marker (add-version skips the stack) and a README with the
  deprecation note.
- hack/migrate-cluster.sh (just migrate-cluster) converts the variables of a
  Cluster manifest from scs2 to the unified variables of scs. Changing the
  ClusterStack of an existing cluster is not possible with the Cluster Stack
  Operator, the script only converts the variables.

Signed-off-by: Jan Schoone <6106846+jschoone@users.noreply.github.com>
Cluster stack for Kubernetes 1.33 to 1.36 that runs the control plane as pods in
the management cluster with the teutonet hosted control plane provider. Only
the workers are OpenStack servers. The API server is exposed through the Gateway
API of the management cluster, Cilium runs on the workers. The ClusterClass is
v1beta1.

The HostedControlPlaneTemplate passes allocate-node-cidrs to the hosted
controller manager. Cilium uses ipam.mode kubernetes and waits for the pod CIDR
of its node. Without the flag the nodes never got a podCIDR, the Cilium agents
did not become ready, the nodes stayed tainted and the cloud controller manager
was never scheduled.

Co-authored-by: Nils Arnold <arnold@aov.de>
Signed-off-by: Jan Schoone <6106846+jschoone@users.noreply.github.com>
…rovider

docker/scs for Kubernetes 1.32 to 1.36 with the Cluster API Docker provider, no
cloud credentials needed. The ClusterClass uses v1beta2, Cilium 1.19.4 and the
metrics-server are applied as addons. Stacks from 1.33 on also ship the Gateway
API CRDs (gateway-api addon).

Signed-off-by: Jan Schoone <6106846+jschoone@users.noreply.github.com>
… stacks

- README and docs/overview.md: the stacks, the just commands and the quickstart.
- docs/versioning.md: layout, requirements (Cluster API >= 1.11, Kubernetes 1.36
  needs >= 1.12.8 or >= 1.13.1, because older versions set the removed kubeadm
  feature gate ControlPlaneKubeletLocalMode), breaking changes only with a new
  minor version, status of the stacks.
- docs/build-system.md: layout, commands, releases and versions, updating, the
  Gateway API CRDs and the migration from scs2 to scs.
- docs/providers: configuration of scs, scs2 and docker/scs (generated sections
  with the variables and the version matrix), hcp with its prerequisites (Gateway
  API and provider version, cert-manager, etcd storage, networkMTU).

Co-authored-by: Nils Arnold <arnold@aov.de>
Signed-off-by: Jan Schoone <6106846+jschoone@users.noreply.github.com>
@jschoone
jschoone requested review from Nils98Ar and garloff October 30, 2026 08:59
image-manager.yaml requires 2048 MiB of RAM per image, but update.sh and
generate-image-manifests.sh still wrote 1024, which would revert the value on
the next update.

Signed-off-by: Nils Arnold <arnold@aov.de>
Otherwise a temporary download error removes the checksum of an unchanged version
from image-manager.yaml.

Signed-off-by: Nils Arnold <arnold@aov.de>
Cilium 1.20 requires it. Without the CRD the operator reports "Required
GatewayAPI resources are not found" and does not enable the Gateway API.

Signed-off-by: Nils Arnold <arnold@aov.de>
The vendored CRDs have to match the Cilium version of the cni addon, otherwise a
Cilium update breaks the Gateway API. update.sh reads the version from the Cilium
release and calls update-gateway-api-crds.sh when it differs.

Signed-off-by: Nils Arnold <arnold@aov.de>
This also moves 1-35 to metrics-server 3.14.0 and to Gateway API v1.6.3, which
Cilium 1.20 needs.

Signed-off-by: Nils Arnold <arnold@aov.de>
…h version

Signed-off-by: Nils Arnold <arnold@aov.de>
The ccm-cloud-config secret lacked settings the old cloud-config secret
had. Add them for both octavia-ovn and octavia-amphora:

- floating-network-id from the networkExternalID variable, needed in
  projects with more than one external network.
- manage-security-groups, now also for amphora, so the amphora can reach
  the NodePorts and the health check node port.
- create-monitor, required for externalTrafficPolicy Local. The CCM uses
  TCP monitors for ovn and HTTP monitors for amphora.

enable-ingress-hostname is left out: it only works with the PROXY
protocol (amphora only), and Cilium honors ipMode Proxy, which the CCM
sets without it.

Signed-off-by: Nils Arnold <arnold@aov.de>
Same version as in 1.35.

Signed-off-by: Nils Arnold <arnold@aov.de>

@Nils98Ar Nils98Ar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good and works for our dev clusters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants