Repository navigation
Let Serverless Workers follow a highly available Namespace through a failover - #4020
Open
rossnelson wants to merge 35 commits into
Open
rossnelson wants to merge 35 commits into
rossnelson wants to merge 35 commits into
Conversation
UI-only mock of Serverless Worker support for multi-region (HA) Namespaces, following the HA PRD. No API changes. - Create Deployment, Create Version and Edit Version forms configure compute per Region (Primary and Replica), with a Serverless toggle per Region, a compact compute provider picker, and Replica fields that can inherit from the Primary - Worker Deployment Preview panel shows where Temporal sends Tasks: Namespace, Worker Deployment and Region nodes on a 12px dot grid, lighting up in Io green as each step is filled in, with a pulse along the active route and a Preview Failover toggle - Deployment detail versions table and expanded row show per-Region compute, provider and connection status; the deployments list shows Region badges next to each version - Failover stories: Failover Readiness Alert, Namespace overview Worker Deployments card and Failover Modal - Region coverage helpers (resolveRegionCoverage, scaling groups with an optional regionId catch-all) and preview connector geometry, with tests - Contrast and ARIA fixes in the Serverless Worker stories Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The summary layout showed only the selected provider and Change, with no hint of what Change offers. A footer now lists the other selectable providers with their logos. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 12, 2026
rossnelson
marked this pull request as draft
October 12, 2026 15:49
Brings 42 commits of main onto the branch, most of which matter because one of them replaced how compute providers are described. The provider registry (COMPUTE_PROVIDERS) now holds each provider's icon, label, description, release stage and infrastructure templates. This branch predates it and hand-rolled all five. The hand-rolled copies are gone and the registry is the source: - compute-provider-picker keeps its summary layout, the Change button and the "also supports" footer, and reads the registry for everything it used to carry itself. It also keeps main's data-release-stage hook, so the badge stays assertable without matching on wording. - compute-fields, the two version forms and the create form take main's single `templates` map in place of the seven per-provider template props. CloudFormation is part of that map now, so the separate CFN props are gone. The region props, the per-region picker and the failover preview are untouched: the UX is the point of this branch.
Two changes to the model under the multi-region UX, both aligning the display side with the form side of this same branch. **A region states its role.** `namespaceRegions` was `readonly string[]` with the primary first by convention, so the display inferred the role from position. The form layer's own `ComputeRegion` already carries an explicit `role`, and an ordering convention is a rule that can break silently where a named role cannot. `NamespaceRegion` now carries it, and a region with no role reads as a replica rather than inventing a primary nobody named. **One question decides whether regions are shown.** The gate is the Namespace's regions, never the compute config. `isMultiRegionConfig` read "any group names a region", which is a question about the config's shape, and answering the first question with the second lets a single-region Namespace with a stray `regionId` read as highly available. It keeps its job under the honest name `hasRegionScopedGroups`: the per-group details table asks it to decide whether its columns have regions to be headed with. The badge now carries `data-region-role` and `data-region-match`, set only when there is more than one region, so the wiring can be asserted on a property rather than on wording. Adds the model's tests and a wiring test that mounts the real page and the real row, so a prop dropped between a page and the badge fails.
rossnelson
force-pushed
the
rossnelson/multi-region-compute
branch
from
October 12, 2026 16:08
b96b02b to
79b97f5
Compare
rossnelson
changed the title
Nothing passed the regions, so nothing multi-region could render against a real server. Self-hosted had the answer all along: a server has no region vocabulary, it has clusters, and the form side already documented that a region id is a cluster id when self-hosted. The active cluster is the primary and the rest hold replicas, which is what the Namespace page already reads out of the same field. The Namespace layout already loads the full DescribeNamespaceResponse, so the two Worker Deployment routes derive the regions from it and pass them down. A Namespace replicated to one cluster, or none, yields fewer than two entries and reads as single region. Also stops this repo parsing a region identifier. `shortRegion` split one on its first hyphen to shorten "aws-us-west-2" for display, which is a guess about a format this repo does not own: it turns a self-hosted cluster named "us-east-1" into "east-1", and naming the clouds explicitly to avoid that would put cloud-specific code here. A region now carries an optional `label`, supplied by whoever supplied the identifier and defaulting to the identifier itself. Matching still joins on the id.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Drives the UI against a multi-cluster deployment so a reviewer can see which regions a Version covers, and the region nothing serves. It starts no server, and that is the finding rather than an omission. `start-dev` names every instance "active", so two of them cannot be registered to each other: the second upsert is refused with "Cannot update current cluster metadata from rpc calls". A Namespace on one cluster is single region, which renders nothing, correctly. The run points the UI at 127.0.0.1:7233, so a multi-cluster frontend goes there. It runs no catalog examples either. The catalog worker is unversioned, so it registers no Worker Deployment and the list it would populate stays empty; and a compute config is attached through the update RPC rather than by a Worker describing itself. Nothing here renders from workflow history. agentcore-compute-provider already sets the same precedent. The notes say what a stock server can and cannot show: without temporalio/api#875 no scaling group carries a region_id, so every group reads as a catch-all and no region reads as uncovered.
temporalio/api#875 merged on 2026-10-08 as `repeated string region_ids`, field 2 on ComputeConfigScalingGroup. This modelled it as a singular scalar and matched on equality, which is wrong on name and on cardinality. A group now serves a region by listing it among others, so matching is containment. A group listing none is the catch-all, as before. The repeated shape is not incidental: it lets one group cover several regions, which is what a provider with a non-regional resource wants. A Lambda ARN is regional, so that case still needs a group per region. Blank entries are dropped rather than read as a region no Namespace has, which is the same care the scalar needed for an empty string. Where no Namespace regions are given and each group speaks for itself, a group naming several regions now shows one row per region, because a row stands for a place workers run. The summary message gained the field too (field 3), so the deployments list can read coverage without fetching each Version. Not used yet.
The save path wrote one group under the literal key `default`, with no region, so a per-region compute config could not be expressed at all. Adds the builder that writes one. A group per region rather than one group listing several: `region_ids` is repeated and would allow the shared shape, but a Lambda ARN names its own region, so two regions cannot share a group even when every other setting matches. A provider whose resource is not regional could use the shared shape later; nothing here prevents it. Each group takes its region's identifier as its name. The name is free text to the API and `region_ids` is what decides where a group runs, but a map keyed by something meaningful reads better in a diff and in an error. Reuses the existing provider builders rather than restating them: each already emits exactly one group, which is the group a region needs.
The ui-server decodes browser JSON into protos with go.temporal.io/api, and v1.63.5 predates region_ids. Its marshaler does not discard unknown fields, so a save carrying regionIds failed as an unknown field, and a read dropped the field before re-encoding, so the browser never saw it. No tagged release contains the field yet; v1.63.6 predates it. This pins the api-go build generated from the proto merge, the same version the server change that maps region_ids uses. The ui-server has pinned pseudo-versions before.
Reading a Namespace's replication clusters as its regions contradicts the server. The worker controller matches region_ids against the region its host supplies, never against cluster names, and the open-source server supplies no host region, so it rejects any group that sets region_ids. The UI would have offered per-region compute that the server refuses. Only a consumer can supply real regions, so the routes pass none outside a demo. A feature demo sets VITE_DEMO_NAMESPACE_REGIONS as regionId:role pairs. The read sits behind import.meta.env.DEV, so a production build never reads it.
The per-region form could not save. No page passed it regions, so it never appeared outside Storybook. Its input lived in local state that never reached submit. And the form validated the single-region fields, which stay empty in multi-region mode, so submit never ran. The per-region component now exposes what each region saves. A replica that is not customized takes the primary's Access and Scaling and keeps its own resource, because the resource is regional. In multi-region mode the form checks only the fields it owns, and each region is checked with the same rules as a single-region Version, with its errors shown in its own block. The create-version and create-deployment pages then build one scaling group per region. The consumer's regions reach the form primary first, because the form reads its first region as the primary and a region states its own role. Removes the replica-only mode, where the customer ran their own Workers in the primary. A multi-region Version runs serverless Workers in every region.
Some features need a change only a demo wants: a value the open-source server leaves out, for example. A scenario can now list patches, which are applied to a checkout this tool fetched before it builds. A checkout somebody pointed at is their working tree and is never patched; the run stops and says so instead. A pinned checkout is reused between runs, so a patch that is already applied is skipped. The patch contents join the binary's cache key, so a changed patch builds again instead of reusing a binary built without it. A scenario can also give the server and the UI dev server extra environment, which is how a UI-side setting such as a VITE_ variable reaches the page.
An existing scaling group changes only in its masked fields, and the mask was always provider,scaler, so an update could never change a group's regions. It is added only when a group sets regions: a server older than region_ids rejects the path, and single-region edits must keep working there.
The scenario now starts its own server instead of expecting one. It builds the branch that maps region_ids into the worker controller, with the worker controller that honours them, and a demo-only patch that gives the server a host region. Without one, the worker controller rejects every scaling group that sets region_ids. Two modules are pinned in the workspace rather than set as floors. The worker controller commit has a pseudo-version that sorts below the release the CLI requires, so a floor loses. The server branch moved to SDK v1.48.0's successor, which removed a field the CLI still sets, so the SDK is held at v1.48.0, which suits both. Pins are a new requirement kind, written as go.work replaces and kept separate from floors, because a pin can move a module backwards. Every port is the scenario's own. A stock server, a dev server without the region variable, or a ui-server built before region_ids would each make the run prove nothing.
Creating a Version validates every group against the provider, so the prerequisite is a Lambda per region and a role the shell can assume, not just credentials. A Version scoped to the primary needs only the primary's Lambda, which is the quickest way to see an uncovered region.
A replica now has a Serverless Workers switch, off unless the caller names the region in enabledRegionIds, for example because the Version being edited already has a group there. Off saves no group for it, says that a failover there finds no workers, and the preview marks the region as missing rather than covered. A server error that names a region shows in that region's block. The worker controller prefixes a group's error with the group's key, and a multi-region Version keys each group by its region, so a missing replica Lambda shows beside the switch that turns it off. An inheriting replica's GCP Project field showed the replica's own empty value while the save used the primary's. It now shows the primary's, and editing it starts the replica's own settings.
A consumer now offers high availability by giving each provider the regions it can run a replica in, as replicaRegions on the provider options it already passes. The list belongs with the consumer, not the provider registry in this repo, because the regions on offer are the consumer's: Temporal Cloud's differ by cloud and by account, and a self-hosted server has none. The form starts single-region. High availability is a switch, off unless the caller names a replica region in enabledRegionIds, and turning it on offers the selected provider's replica regions to pick one from. Off saves one group that no region scopes, exactly as a single-region Version does, so the pages need no new path for it. On saves a group for the primary and one for the chosen replica. Changing the replica region starts its resource over, because a resource is regional. The settings that stand in for a consumer during development are named for what they are rather than for the demo: VITE_NAMESPACE_REGIONS and VITE_REPLICA_REGIONS, read in env-regions.ts.
- An ARN takes a region's label, the provider's own name for it, and only when a consumer gave one. Falling back to the id built invalid ARNs such as arn:aws:lambda:aws-us-east-1:…, as both the placeholder and the replica's suggestion. Without a label, nothing is suggested. - Each region badge names the provider of the group that serves that region. An uncovered region names none; it used to claim the Version's first provider, as if Lambda served a region nothing serves. - The heading says where Serverless Workers run, not what the Namespace is. Turning high availability on does not change the Namespace. It also stops stripping Temporal Cloud's account suffix from the Namespace name. - With more than two other providers, the picker's footer counts them instead of listing every name; the icons already show which. - The development region settings accept regionId=label and offer only the providers they name, as a consumer offers only its own cloud's.
Without high availability the form saves one group that no region scopes, so labelling its card with the Namespace's region claimed something the config does not say, whichever provider was picked. The card and heading name regions only when high availability scopes them. Every provider is offered again. Hiding the ones without replica regions removed choices to hide a mislabelled card.
The switch sat in a card of its own above the replica's. The replica's card now carries it in its header: off, the card reads Replica and says what happens without one; on, it names the chosen region and shows the region select and the replica's fields. A region card can take a title for the state with no region to name.
With high availability on, the primary saved a group scoped to the Namespace's primary region. The PRD's upgrade path keeps the existing default group and adds one scoped to the replica: the default runs wherever no scoped group matches, as every group did before regions, and turning high availability on only ever adds a group. The release tracker gives the same steps. So the primary's only region is the one its resource names, and its card no longer names a region in either state. Text that named the primary by region now says the primary. An error on the default group shows on the primary's card. The Region heading stays on the page with high availability off, with a line of the same height, so turning it on changes the line instead of inserting a heading and moving the page.
The replica's region comes from its resource, matched against the consumer's labels, and the region select is gone. Headers name a region once a resource gives one. Removes the Preview Failover button. Not pushed: the platform does not limit a resource's region, so this may be reverted in favour of a select with a mismatch warning.
The worker controller picks a scaling group by matching its own region, not by trying one resource and falling back to another, so the replica's group must name the region the Namespace fails over to. That region is not a choice: it is the Namespace's replica. So there is no region to select and nothing to infer from a resource. Both entry points have one. Self-hosted reads it from the Namespace's replication config again, the active cluster being the primary; Temporal Cloud passes its regions. A development setting stands in for either when the server cannot provide them. High availability is offered when the Namespace has a replica. A resource's region stays free, as it is today: neither the UI nor the worker controller limits it. Putting the primary and the replica in one region draws a warning, not an error, because an outage there stops both. Removes the per-provider replica regions and their development setting, which only a region select needed.
The primary's header named the region in its ARN and the replica's named the Namespace's replica, so a replica pointed at us-east-1 read us-west-2 above a warning about us-east-1. Both headers now name where the resource runs, from its ARN, and read Primary and Replica until there is one. The replica's group is still scoped to the Namespace's replica; the Regions line and the Replica tooltip say which. Turns off spellcheck on ARN fields, which underlined every ARN.
region_ids names the Namespace regions the worker controller runs in, and the controller acts only with a host region. So with high availability on the headers show those regions, from the Namespace config: the primary and the replica the group is scoped to. Off, they read Primary and Replica. A resource's own region no longer changes them; it only feeds the same-region warning.
rossnelson
changed the title
The edit form reads a saved config the way the create form writes one: the unscoped group fills the primary card, and the group scoped to the Namespace's replica fills the replica card. High availability starts on when that group exists. A replica whose Access or Scaling differs from the primary's starts customized, so an edit does not quietly replace its settings with the primary's. Saving keeps each group's name, so a replica saved as "secondary" is updated in place rather than joined by a second group. Turning high availability off removes the replica's group through remove_compute_config_scaling_groups. Only the groups this form edits are ever removed. Also fixes the region_ids update mask. A mask sent as JSON names fields in lowerCamelCase, so the path is regionIds; region_ids was rejected as an invalid path. Verified against the demo server: turning high availability off removed the replica group and kept the default one, turning it on added a group scoped to the replica, and editing a Version with a "secondary" group kept that name.
The expanded Version details named a group by its raw region ids, and the unscoped group as All Regions. Pass the Namespace's regions through and label each column the way the form does: the unscoped group as the primary, a scoped group as its region and role. Align the column headers with their values, and inset the cells so the row dividers extend past the text.
Each animation shows a change the user caused. A reveal grows down from the control they used, and a swap fades in, so nothing above the change moves. All durations are 150-200ms and drop to 0 when the user prefers reduced motion. - Provider picker: the cards slide open from Change, and the summary fades back. The fields below fade when the provider changes, because their labels change. - High availability: the replica body slides open and closed under the toggle, and the region names fade into both card headers. - Customize: the replica's Access and Scaling slide open. - Same-region warning: slides in after the ARN stops changing for 400ms, so it does not flicker while the user types. It sits in a live region. - The suggested-ARN hint and a new preview card fade in. The preview pulse plays twice when the regions change, then stops. Focus and layout fixes the motion depends on: - The picker moves focus to the checked card when it opens and back to Change when it closes. Arrow keys browse the cards without closing them; a click, Enter, or Space chooses. - Customize moves focus to the first field it reveals. - On submit, the form focuses the first invalid region field, or scrolls to a server error, so errors above the viewport are not missed. - The high availability sentence is now shorter than the single-region one, so turning the toggle on cannot wrap it and move the toggle.
Closing the cards inside the click handler removed the radio before the click could select it, so the provider did not change. Close when the value changes instead, and keep the cards open only for a change an arrow key made. A click on the card already chosen closes them directly.
…yment pages - The five worker deployment pages take one providerConfig prop instead of computeProviders, gcpRegions and namespaceRegions. It holds the providers a Namespace may use and the regions it lives in; two or more regions turn on high availability. Cloud Run's deploy regions move onto its provider. - Self-hosted routes no longer read regions from the Namespace's replication config. The server has no host region for region_ids to match, so they offer no high availability; VITE_NAMESPACE_REGIONS still stands in during development.
The Forms and Pages stories now define a ComputeProviderConfig per Namespace and derive what each form or page receives from it with the same helpers the pages use, so they show the shape a host passes.
Main moved the demo definitions to src/lib/demo, so point the multi-region scenario at the new path.
Contributor
Two things leaked the feature into builds that can never use it.
The routes read Namespace regions from a dev-only environment variable, so
five production route files imported a helper whose first line is
`if (!import.meta.env.DEV) return undefined`. In production all five then
passed `{ regions: undefined }`, which is what passing nothing already means.
It bought no behaviour and cost a public env var, a bespoke `region:role`
parser, its tests, and the imports. import.meta.env.DEV appears nowhere else
in src/, and env.d.ts held only two build-wide vars, so this introduced a
pattern rather than following one. The components still demo through their
stories.
RegionBlock rendered its card unconditionally. The header was correctly
hidden with no region to name, but the border and padding were not, so every
single-region form gained a card it never had — visible to everyone the
moment this packs, with the feature off. The card is the region boundary, so
with no region there is nothing to bound and the children render bare.
A consumer such as Temporal Cloud still turns the feature on by passing
providerConfig.regions; nothing else can.
…on-fixes # Conflicts: # server/go.mod # server/go.sum
rossnelson
marked this pull request as ready for review
October 23, 2026 20:56
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets a Serverless Worker Deployment Version follow a highly available Namespace through a failover, and shows which regions each Version covers.
Built on @bilal-karim's design prototype in #3951; his commits are preserved here.
How
region_idsworksregion_idson a scaling group (temporalio/api#875) names the Namespace regions the worker controller runs in. It does not say where the function runs. The controller learns its own region from its host and uses the groups that list it. A group with no regions is the default group, used wherever no scoped group matches. A function's region comes only from its ARN, and nothing limits it.The form
High availability is offered only when the Namespace has a replica, and it starts off, because HA for compute is opt-in per Version.
The replica card inherits the primary's Access and Scaling until it is customized, and suggests its ARN from the primary's by swapping the region. With HA on, the card headers name the Namespace's regions. A warning shows when both functions run in one region, because an outage there stops both; it does not block a save. A save error shows in the card of the group it names.
Editing
The edit form reads a saved config the way the create form writes one. The unscoped group fills the primary card, and the group scoped to the Namespace's replica fills the replica card. HA starts on when that group exists. A replica with its own Access or Scaling starts customized. Saving keeps each group's name, and turning HA off removes the replica's group through
remove_compute_config_scaling_groups. Only the groups the form edits are ever removed.Where the regions come from
The worker deployment pages take one prop,
providerConfig, built once per Namespace by whoever hosts them:It replaces
computeProviders,gcpRegionsandnamespaceRegions. The Namespace's regions sit at the top level because they are one fact about the Namespace, not about a provider, and the deployment list pages need them without providers. A fact about one provider goes on that provider: Cloud Run's GCP regions are now itsdeployRegions.regions, each with a label.region_idsagainst its host region, and the open-source server supplies none (see Map region_ids between compute config and WCI scaling group spec temporal#12491).This repo does not parse a region identifier. A consumer gives each region a label, and the label is used for display and to suggest a regional ARN.
Coverage
The Version rows and the deployments list show the regions a Version covers. A region that only the default group covers is shown that way, and a region that nothing serves is the one that is coloured.
Supporting changes
go.temporal.io/api v1.64.0, the first release carryingregion_ids. Without it the ui-server rejects the field on save and drops it on read, so the earlier pre-release pin is gone.regionIdsfor a group that sets it, so a group's regions can change. Single-region updates keep their old mask, so they still work on older servers.Feature demo
This builds the server from the branch that maps
region_ids(temporalio/temporal#12491) with the worker controller change (temporalio/temporal-auto-scaled-workers#152). It also applies a demo-only patch that gives the server a host region, because the open-source server supplies none, and without one the worker controller rejects groups that setregion_ids. The demo harness can now patch a fetched checkout, pin workspace modules, and pass environment to the server and the UI.Saving validates each group against AWS, so it needs a Lambda in each region and credentials that can reach them.
Not in this PR
Testing
pnpm checkreports 0 errors and all three linters pass. 4,033 unit tests pass; the only failures are thescripts/catalogtests, which shell out to a nestedpnpmand so fail on any Node below the^22.23.4this repo now requires. They pass in CI. Against the demo server, a region-scoped Version saves and reads back itsregion_idson the Version, the version summaries, and the deployments list. On the edit page, turning HA off removed the replica group and kept the default one, turning it on added a group scoped to the replica, and a replica group namedsecondarykept its name.Reviewing this
It is a wide diff, so the commits are kept clean and the quickest route is to read them in order rather than the combined diff.
The high availability machinery is the part nobody can see yet. It turns on only when a consumer passes
providerConfig.regionswith a replica, and no consumer does:src/lib/utilities/compute-regions.tsandcompute-provider-config.ts- the model and the gatecompute-provider-per-region.svelte- the two-card form and the HA togglefailover-preview.svelte,regions-heading.svelte,region-block.svelteoffersHighAvailabilitychooses the branchThe changes everyone sees are smaller, and worth separate attention because they ship whether or not high availability is ever switched on:
version-regions.sveltereplacescompute-badge.sveltein the deployments table: the provider pill becomes a rounded two-tone chip with the short label. With no Namespace regions it falls back to the Version's own scaling groups, which today means one unscoped group, and so no region names.compute-provider-picker.sveltegained a summary footer naming the other providers.A consumer has previewed the whole pack with high availability switched off, and the worker deployment pages render as they do today.