Repository navigation
Conversation
lwille
commented
Nov 9, 2026
|
|
||
| for server_type_index in $(seq 0 $((server_type_count - 1))); do | ||
| for location_index in $(seq 0 $((location_count - 1))); do | ||
| for location_index in $(seq 0 $((location_count - 1))); do |
Contributor
Author
There was a problem hiding this comment.
Hetzner removed the Datacenters API on 2026-10-01; requests now return HTTP 410 (deprecated_api_endpoint), breaking every CD plan. Switch the location pre-flight check to the hcloud_locations data source and pin the hcloud provider to ~> 1.70 so provider upgrades are deliberate.
The retry loop iterated locations inside server types, so a capacity miss for the first server type in hel1 fell through to fsn1/nbg1, where VMs are rarely available. Iterate server types inside locations so hel1 is exhausted first.
The cleanup steps ran with `if: always()`, so when an earlier step (e.g. the dev build) failed they ran before setup-terraform and died with `terraform: command not found`. Only run them once terraform was installed; if it wasn't, nothing can have been deployed.
lwille
force-pushed
the
CO-38/fix-deployment-pipeline
branch
from
November 9, 2026 10:27
c393e5e to
2a5070d
Compare
6 of 12 tasks
setup_nodes.yml connected to adminhost_local (private IP, through the public adminhost) right after the VMs were created and failed with 'Connection timed out during banner exchange'. Wait for SSH first, as #919 does for wiab-staging.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Since 2026-10-01 Hetzner Cloud returns
HTTP 410 Gone(deprecated_api_endpoint) for the Datacenters API (changelog). All three CD terraform modules readdata "hcloud_datacenters"for the location pre-flight check, so everyterraform planfails and all retries incd*.share exhausted (e.g. https://github.com/wireapp/wire-server-deploy/actions/runs/38375138298).Ticket: https://wearezeta.atlassian.net/browse/CO-38
Changes
wiab-demo-hetzner,wiab-staging-hetzner,wire-server-deploy-offline-hetzner: replacehcloud_datacenterswithhcloud_locations; location validation logic unchanged.hetznercloud/hcloudto~> 1.70(previously unconstrained, CI always pulled latest).offline/cd.sh,cd_demo.sh,cd_staging.sh: iterate server types inside locations, so all server types are tried inhel1before falling back to the German locations (which rarely have capacity).offline.yml: only run the Hetzner cleanup steps if terraform was installed; previously a failed build made cleanup fail withterraform: command not found.wire-server-deploy-offline-hetzner/setup_nodes.yml: wait for SSH on the adminhost (public, then private IP) before the first play, as fix: WPB-25755 fix for workflows when pulling images and connecting to new VMs #919 does for wiab-staging. Without it the default verify can fail right after VM creation withConnection timed out during banner exchangeonadminhost_local(seen twice in https://github.com/wireapp/wire-server-deploy/actions/runs/42643467944).changelog.d/5-bug-fixes/.The MinIO mirror fix needed for the build to pass is on master via #943; this PR is rebased onto it.
Testing
terraform validatepasses for all three modules (provider 1.70.2).build-allrun https://github.com/wireapp/wire-server-deploy/actions/runs/42073445002: all jobs green (dev, min, default build; default + wiab-staging Hetzner verify; container). All deployments landed inhel1.scpconnection pre-auth (Connection closed by ... port 22). The re-run passed; looks transient, unrelated to this change.