Repository navigation
Conversation
Hetzner removed the Datacenters API on 2026-10-01; requests now return HTTP 410 (deprecated_api_endpoint), breaking every CD plan. Switch the location pre-flight check to the hcloud_locations data source and pin the hcloud provider to ~> 1.70 so provider upgrades are deliberate.
The retry loop iterated locations inside server types, so a capacity miss for the first server type in hel1 fell through to fsn1/nbg1, where VMs are rarely available. Iterate server types inside locations so hel1 is exhausted first.
The cleanup steps ran with `if: always()`, so when an earlier step (e.g. the dev build) failed they ran before setup-terraform and died with `terraform: command not found`. Only run them once terraform was installed; if it wasn't, nothing can have been deployed.
The ingress-nginx-controller chart of wire-server 5.25.0 bundles ingress-nginx 4.11.5, and the values pin controller image v1.10.6, which has known CVEs. - Replace the ingress-nginx subchart with 4.13.9 during the bundle build (pre_chart_process_0.sh). 4.13.x is the last ingress-nginx line that supports Kubernetes 1.29, which this bundle installs. - Pin controller v1.13.9 and kube-webhook-certgen v1.6.9 in the values examples (tags only, digests are not supported by the bundle tooling). - Set annotations-risk-level: Critical. Controller >= 1.12 defaults to High and rejects the nginx-ingress-services ingresses that use server-snippet / configuration-snippet annotations. - Add an upgrade guide for existing installations and changelog entries.
* WPB-29054: use minio mirror for fake-aws-s3 Porting: > https://wearezeta.atlassian.net/browse/WPB-29054 > Note: Fixing or replacing the broken fake-aws-s3 chart is not in scope. * lint: add newline at end of file * lint: newline * fix: block fake-aws-s3 from default-build * removed ineffective values
setup_nodes.yml connected to adminhost_local (private IP, through the public adminhost) right after the VMs were created and failed with 'Connection timed out during banner exchange'. Wait for SSH first, as #919 does for wiab-staging.
lwille
added
build-dev
build-wiab-staging
and removed
build-wiab-staging
build-dev
labels
release-5.25-R2 commented out `main` (846d27f), and only cd_staging.sh appended it back on the CI host. The default verify (cd.sh) therefore only installed Kubernetes and never deployed any chart, so it did not test the ingress-nginx upgrade. Re-enable `main` and drop the append from cd_staging.sh (it would now run main twice). The upgrade guide warns not to run offline-deploy.sh / helm-operations.sh on existing installations, as they replace values.yaml with the examples.
lwille
marked this pull request as draft
November 11, 2026 15:25
mohitrajain
approved these changes
Nov 12, 2026
|
|
||
| # pulling the charts based on builds.json, charts to be skipped are passed as arguments HELM_CHART_EXCLUDE_LIST | ||
| "${TASKS_DIR}"/proc_pull_charts.sh OUTPUT_DIR="${OUTPUT_DIR}" HELM_CHART_EXCLUDE_LIST="inbucket,wire-server-enterprise,postgresql,rust-sft,fluent-bit" | ||
| "${TASKS_DIR}"/proc_pull_charts.sh OUTPUT_DIR="${OUTPUT_DIR}" HELM_CHART_EXCLUDE_LIST="inbucket,wire-server-enterprise,postgresql,rust-sft,fluent-bit,fake-aws-s3" |
Contributor
There was a problem hiding this comment.
Maybe we can also add nginx-ingress-controller to the list - we don't use it and it is coming from builds.json.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes https://wearezeta.atlassian.net/browse/WPB-26631
Change type
Basic information
Testing
Locally verified: running
pre_chart_process_0.shagainst the 5.25.0ingress-nginx-controllerchart vendors ingress-nginx 4.13.9;helm templatewith the prod and demo values renderscontroller:v1.13.9,kube-webhook-certgen:v1.6.9(tags, no digests) andannotations-risk-level: "Critical";helm lintandshellcheckpass.Offline Build CI (label-based)
build-default: builds the bundle and deploys it to Hetzner viaoffline/cd.sh. The deployment uses the prod example values, so it exercises theserver-snippetannotation on the minio ingress ofnginx-ingress-services.Tracking
changelog.dKnowledge Transfer
* [ ] An Asciinema session is attached to the Jira ticket.Motivation
A customer on 5.25 reported CVEs in the bundled ingress-nginx controller image (
v1.10.6, pinned invalues/ingress-nginx-controller). Bumping the image tag tov1.12.1brokenginx-ingress-serviceswith:This branch provides a patched 5.25 bundle for that customer. It is not an official release.
Objective
ingress-nginx-controllercontroller.config.annotations-risk-levelNothing else changes; wire-server stays on 5.25.0.
Reason
kube_version: v1.29.10in this bundle). 4.14 needs 1.30, 4.15 needs 1.31.pre_chart_process_0.shinstead of changing wire-builds, so only this bundle is affected.annotations-risk-leveltoHigh.server-snippetandconfiguration-snippet(minio, nginz and federator ingresses) areCritical, so the setting has to beCritical. It belongs to the ingress-nginx-controller release, not tonginx-ingress-services.enableAnnotationValidations: false(used in WPB-26631: ingress-nginx-controller wire-server#5623) is not needed: it works around Wrong validation for permanent-redirect and temporal-redirect annotation values kubernetes/ingress-nginx#12709 (redirect annotations), which the 5.25.0 charts don't use.quay.io/minio/mcimage is no longer pullable, which breaks the bundle build. Uses the Wire mirror for the minio images and dropsfake-aws-s3from the default build (prod uses an external minio).Use case
Existing installations must update their own
values/ingress-nginx-controller/values.yaml(image tags andannotations-risk-level: Critical) and upgradeingress-nginx-controllerbeforenginx-ingress-services. Step-by-step guide:offline/upgrades/ingress-nginx-controller-v1.13.md.